Also on LinkedIn
Loading...
Flash Player 9 (or above) is needed to view slideshows. We have detected that you do not have it on your computer.To install it, go here
-
locifer favorited this 2 weeks ago
SlideShare is now available on LinkedIn. Add it to your LinkedIn profile.
Amazon S3 storage engine plugin for MySQL
1214 views | 0 comments | 1 favorites | 23 downloads | embeds (Stats)
Slideshow Transcript
- Slide 1: A Storage Engine for Amazon S3 How to store a hundred billion BLOBs without buying a single disk.
- Slide 2: Who is this guy?
- Slide 3: http://fallenpegasus.com/code
- Slide 4: Storage Engine
- Slide 5: Protocol Translator
- Slide 6: Traditional Storage Engines use the local disk
- Slide 7: Network Storage Engines
- Slide 8: Federated ODBC HTTP MemCacheD and...
- Slide 9: S3
- Slide 10: What is S3?
- Slide 11: Petabytes?!
- Slide 12: This can't be free
- Slide 13: Why do I want to use it?
- Slide 14: “An empty disk costs the same as a full one.”
- Slide 15: Can I move my existing database over?
- Slide 16: Translating S3 to MySQL AWS Account → CREATE SERVER • S3 Bucket → Table • S3 Item → Row • S3 Item Key → VARCHAR PRIMARY KEY • S3 Item Contents → BLOB or VARCHAR •
- Slide 17: CREATE SERVER 'MyAWSAcct' FOREIGN DATA WRAPPER 'AWS' OPTIONS (USER 'aws id string', PASSWORD 'aws secret string');
- Slide 18: CREATE TABLE 'bierce' ( 'word' VARCHAR(255) NOT NULL PRIMARY KEY, 'defn' BLOB) CHARSET=utf-8 ENGINE=AWSS3 CONNECTION='awss3 DevilDictionary $server MyAWSAcct';
- Slide 19: SELECT defn FROM bierce WHERE word='WIT';
- Slide 20: SELECT defn FROM bierce WHERE word='WIT'; WIT, n. The salt with which the American humorist spoils his intellectual cookery by leaving it out.
- Slide 21: INSERT INTO bierce (word, defn) VALUES ('AUTHOR', 'One noted for confusing bitterness with humor.');
- Slide 22: DELETE FROM bierce WHERE word='AUTHOR';
- Slide 23: So, what's it good for?
- Slide 24: “If you build it, they will come.”
- Slide 25: Saving EC2 work • SQL CMS for S3 virtual hosting • Huge list of persistent primary keys • Big slow BLOBs, to join against fast local tables. • “The Image Server Problem” • ... and more •
- Slide 26: Transactions?
- Slide 27: What do you mean, “No temporal guarantees”?!
- Slide 28: No WHERE clause means “MAKE MONEY FAST” for Amazon
- Slide 29: The Future (watch me wave my hands)
- Slide 30: Code Improvement • S3 & HTTP Metadata • Multiple Data Columns • Information Schema • Security & Authentication • Transfer & Storage Compression • Streaming BLOB • More Storage Engines •
- Slide 31: Code Improvement
- Slide 32: S3 & HTTP Metadata
- Slide 33: Multiple Data Columns
- Slide 34: Sharing a solution with HTTP and MemCacheD Engines
- Slide 35: Information Schema
- Slide 36: Security & Authentication
- Slide 37: Transfer & Storage Compression
- Slide 38: (new slide) Streaming BLOBs like PBXT
- Slide 39: More storage engines
- Slide 40: More storage engines “A Storage Engine for AWS SQS” • There are more AWS services rolling out • Replication distribution via S3 • Cluster persistent storage via S3 • ... and the EC2 holy grail: •
- Slide 41: “A Generic Schema Storage Engine for S3”
- Slide 42: Back to the present...
- Slide 43: “Where do I get it?”
- Slide 44: “Where do I get it?” MySQL 5.1 • GPL • http://fallenpegasus.com/code •
- Slide 45: “You never write, you never call...”
- Slide 46: Suggestions • Use Cases • Bug Reports • Patches • Feature Requests • Support Requests • Gigs & Proposals • mark@fallenpegasus.com
- Slide 47: Questions?
