Amazon Elastic MapReduce (Amazon EMR) ã¯ã倧éã®ããŒã¿ãè¿ éã容æã«ããã€ã³ã¹ã广ããåŠçããããã®ãŠã§ããµãŒãã¹ã§ãã
Amazon EMR ã¯ãããã°ããŒã¿åŠçãç°¡ç¥åããåçã«ã¹ã±ãŒã©ãã«ãª Amazon EC2 ã€ã³ã¹ã¿ã³ã¹éã®è«å€§ãªéã®ããŒã¿ãåŠçããé ä¿¡ããããã®ãç°¡åãè¿ éãè²»çšå¯Ÿå¹æãé«ãããããŒãžãå Hadoop ãã¬ãŒã ã¯ãŒã¯ãæäŸããŸãããŸããSpark ã Presto ãªã©ã®äžè¬çãªä»ã®ãã¬ãŒã ã¯ãŒã¯ã Amazon EMR ã§å®è¡ããããAmazon S3 ã Amazon DynamoDB ãªã©ã®ä»ã® AWS ããŒã¿ã¹ãã¢å ã§ããŒã¿ãæäœã§ããŸãã
Amazon EMR ã¯ããã°ã®åæããŠã§ãã€ã³ããã¯ã¹ãããŒã¿ãŠã§ã¢ããŠãžã³ã°ãæ©æ¢°åŠç¿ã財ååæãç§åŠã·ãã¥ã¬ãŒã·ã§ã³ãçç©æ
å ±ç§åŠç ç©¶ãå«ããã客æ§ã®ããã°ããŒã¿ã確å®ãã€å®å
šã«åŠçããŸãã
AWS ãç¡æã§ã詊ããã ãã
ãŸãã¯ç¡æã§å§ãã »ãŸãã¯ã³ã³ãœãŒã«ã«ãµã€ã³ã€ã³
å€§èŠæš¡ãªããŒã¿åŠçèŠä»¶ãããå Žåã¯ãå²åŒæéãé©çšãããå¯èœæ§ããããŸãã
詳现ã«ã€ããŠã¯ããåãåãããã ããã
BlueTalon ã®åµèšè
å
Œè£œåæ
åœè²¬ä»»è
Pratik Verma æ°ã«ããã²ã¹ãæçš¿èšäº BlueTalon ã Amazon EMR ã§äœ¿çšããã
Amazon VPC ç°å¢å ã§ Amazon EMR ã¯ã©ã¹ã¿ãŒãèµ·åããå®è¡ããæ¹æ³ã«ã€ããŠç¢ºèªãã
Amazon EMR ã¯ã©ã¹ã¿ãŒã¯æ°åã§èµ·åã§ããŸããããŒãã®ããããžã§ãã³ã°ãã¯ã©ã¹ã¿ãŒã®ã»ããã¢ãããHadoop ã®èšå®ãã¯ã©ã¹ã¿ãŒã®åŸ®èª¿æŽãªã©ãããããããäœæ¥ã¯äžèŠã§ããAmazon EMR ããã®ãããªäœæ¥ãè¡ããŸãã®ã§ãã客æ§ã¯åæã«å°å¿µã§ããŸãã
Amazon EMR ã®æéäœç³»ã¯äºæ³ãããããã·ã³ãã«: 䜿çšããã€ã³ã¹ã¿ã³ã¹ã®æéããšã« 1 æéåäœã§æéãæ¯æããŸãã10 ããŒãã® Hadoop ã¯ã©ã¹ã¿ãŒãããã 0.15 USD/æéãšããäœæéã§éçšã§ããŸããAmazon EMR 㯠Amazon EC2 ã®ã¹ãããã€ã³ã¹ã¿ã³ã¹ãšãªã¶ãŒããã€ã³ã¹ã¿ã³ã¹ããã€ãã£ããµããŒãããŠãããããéçšã«äœ¿ã EC2 ã€ã³ã¹ã¿ã³ã¹ã®ã³ã¹ãã 50ïœ80% ç¯çŽããããšãå¯èœã§ãã
Amazon EMR ã¯ãã³ã³ãã¥ãŒãã£ã³ã°ã€ã³ã¹ã¿ã³ã¹ã 1 ã€ã§ãæ°çŸãæ°åã§ãããããžã§ãã³ã°ã§ãããããããããèŠæš¡ã®ããŒã¿åŠçã«å¯Ÿå¿ããŸããã€ã³ã¹ã¿ã³ã¹æ°ã¯å®¹æã«å¢æžã§ãããæ¯æãã¯äœ¿ã£ãåã ãã®åŸéæéå¶ã§ãã
ã¯ã©ã¹ã¿ãŒã®èª¿æŽãã¢ãã¿ãªã³ã°ã«ãããåŽåãåæžã§ããŸããAmazon EMR ã«ã¯ã¯ã©ãŠãçšã®èª¿æŽããã Hadoop ããããã¯ã©ã¹ã¿ãŒãã¢ãã¿ãªã³ã°ããŠå€±æããã¿ã¹ã¯ãå詊è¡ããèªåçã«ããã©ãŒãã³ã¹ã®äœãã€ã³ã¹ã¿ã³ã¹ã眮ãæããŸãã
Amazon EMR 㯠Amazon EC2 ã€ã³ã¹ã¿ã³ã¹ãžã®ãããã¯ãŒã¯ã¢ã¯ã»ã¹ãå¶åŸ¡ãããã¡ã€ã¢ãŠã©ãŒã«ã®åçš®èšå®ãèªåçã«æ§æããŸãããŸããã客æ§ãå®çŸ©ããè«ççã«éé¢ããããããã¯ãŒã¯ã§ãã Amazon Virtual Private CloudïŒVPCïŒå
ã«ã¯ã©ã¹ã¿ãŒãèµ·åããããšãå¯èœã§ããAmazon S3 ã«ä¿åããããªããžã§ã¯ãã®å ŽåãAWS Key Management Service ãŸãã¯ã«ã¹ã¿ããŒç®¡çåã®ããŒã䜿çšã㊠Amazon S3 ãµãŒããŒåŽã®æå·åãŸã㯠Amazon S3 ã¯ã©ã€ã¢ã³ãåŽã®æå·åãš EMRFS ã䜿çšã§ããŸãã
ã客æ§ãã¯ã©ã¹ã¿ãŒã®ãã¹ãŠãå¶åŸ¡ããããšãã§ããŸããã客æ§ã¯åã€ã³ã¹ã¿ã³ã¹ãžã®ã«ãŒãã¢ã¯ã»ã¹æš©éãæã¡ãç°¡åã«ãã®ä»ã®ã¢ããªã±ãŒã·ã§ã³ã远å ããã¯ã©ã¹ã¿ãŒããšã«èšå®ãã«ã¹ã¿ãã€ãºã§ããŸãããŸã Amazon EMR ã¯ãè€æ°ã® Hadoop ãã£ã¹ããªãã¥ãŒã·ã§ã³ãšã¢ããªã±ãŒã·ã§ã³ã«å¯Ÿå¿ããŠããŸãã
Amazon EMR ã¯ã¯ãªãã¯ã¹ããªãŒã ããŒã¿ãåæããŠããŠãŒã¶ãŒãåºåããããŠãŒã¶ãŒã®å¥œã¿ãçè§£ãããããç®çã«ã䜿çšã§ããŸããåºåäž»ãã¯ãªãã¯ã¹ããªãŒã ã宣äŒã®ã€ã³ãã¬ãã·ã§ã³ãã°ãåæããããšã§ããã广çãªå®£äŒã®é
ä¿¡ã«æŽ»çšã§ããŸãã
Razorfish ã«ãã EMR ã䜿ã£ãã¯ãªãã¯ã¹ããªãŒã ã®åæäºäŸã®è©³çް »
Amazon EMR ã§ã¯ãèšå€§ãªéã®ã²ããã¯ã¹ããŒã¿ãããã®ä»ã®å€§éã®ç§åŠããŒã¿ã»ããããã°ããå¹ççã«åŠçããããšãã§ããŸããç ç©¶è
㯠AWS ã§ç¡æã§ãã¹ããããŠããã²ããã¯ã¹ããŒã¿ã«ã¢ã¯ã»ã¹ã§ããŸãã
1000 Genomes ãããžã§ã¯ããš AWS ã«é¢ããèšäºãèªã »
Amazon EMR ã¯ãŠã§ããã¢ãã€ã«ã¢ããªã±ãŒã·ã§ã³ã§çæããããã°ã®åŠçã«ã䜿çšã§ããŸããAmazon EMR ã䜿çšãããšããã¿ãã€ãçŽã®éæ§é åããŒã¿ãåæ§é åããŒã¿ããã¢ããªã±ãŒã·ã§ã³ããŠãŒã¶ãŒã«é¢ããæçãªæŽå¯ãåŸãããšãå¯èœã«ãªããŸãã
Yelp ã EMR ã䜿çšããŠäž»ãªãŠã§ããµã€ãæ©èœãæšé²ããæ¹æ³ »
Amazon EMR ã䜿çšããæé ã¯æ¬¡ã®ãšããã§ãã
- ããŒã¿åŠçã¢ããªã±ãŒã·ã§ã³ãéçºãããJavaãHiveïŒSQL ã«é¡äŒŒããèšèªïŒãPigïŒããŒã¿åŠçèšèªïŒãCascadingãRubyãPerlãPythonãRãPHPãC++ãNode.js ã䜿çšã§ããŸããAmazon EMR ã«ã¯ã³ãŒããµã³ãã«ãšãã¥ãŒããªã¢ã«ãçšæãããŠãããããã«äœ¿çšãéå§ã§ããŸãã
- ã¢ããªã±ãŒã·ã§ã³ãšããŒã¿ã Amazon S3 ã«ã¢ããããŒãããã倧éã®ããŒã¿ãã¢ããããŒãããå Žåã¯ãAWS Import/ExportïŒç©çèšæ¶è£ 眮ã䜿çšããããŒã¿ã®ã¢ããããŒãïŒãŸã㯠AWS Direct ConnectïŒããŒã¿ã»ã³ã¿ãŒãã AWS ãžã®å°çšã®ãããã¯ãŒã¯æ¥ç¶ã確ç«ïŒã®äœ¿çšãæ€èšããŠãã ããããåžæã§ããã°ãå®è¡äžã®ã¯ã©ã¹ã¿ãŒãžçŽæ¥ããŒã¿ãæžã蟌ãããšãã§ããŸãã
- ã¯ã©ã¹ã¿ãŒã®èšå®ãšèµ·åãAWS ãããžã¡ã³ãã³ã³ãœãŒã«ãAWS CLIãSDK ãŸã㯠API ã䜿çšããŠãã¯ã©ã¹ã¿ãŒã«æäŸãã EC2 ã€ã³ã¹ã¿ã³ã¹ã®æ°ã䜿çšããã€ã³ã¹ã¿ã³ã¹ã¿ã€ãïŒã¹ã¿ã³ããŒãããã€ã¡ã¢ãªãã〠CPUãã〠I/O ãªã©ïŒãã€ã³ã¹ããŒã«ããã¢ããªã±ãŒã·ã§ã³ïŒHiveãPigãHBase ãªã©ïŒãã¢ããªã±ãŒã·ã§ã³ãšããŒã¿ã®å Žæãæå®ããŸããããŒãã¹ãã©ããã¢ã¯ã·ã§ã³ã䜿çšããŠè¿œå ã®ãœãããŠã§ã¢ãã€ã³ã¹ããŒã«ããããæ¢å®ã®èšå®ã倿Žãããã§ããŸãã
- ïŒãªãã·ã§ã³ïŒã¯ã©ã¹ã¿ãŒãç£èŠããããããžã¡ã³ãã³ã³ãœãŒã«ãã³ãã³ãã©ã€ã³ã€ã³ã¿ãŒãã§ã€ã¹ãSDKããŸã㯠API ã䜿çšããŠã¯ã©ã¹ã¿ãŒã®å¥å šæ§ãšé²è¡ç¶æ³ãç£èŠã§ããŸããEMR 㯠Amazon CloudWatch ãšçµ±åããŠã¢ãã¿ãªã³ã°/èŠåãè¡ããGanglia ã®ãããªæåãªã¢ãã¿ãªã³ã°ããŒã«ããµããŒãããŠããŸããåŠçããããŒã¿ã®éã«å¿ããŠãã¯ã©ã¹ã¿ãŒã®èœåã®è¿œå ãšåé€ã¯ãã€ã§ãè¡ãããšãã§ããŸãããã©ãã«ã·ã¥ãŒãã£ã³ã°ã«ã¯ãã³ã³ãœãŒã«ã®ã·ã³ãã«ãªãããã° GUI ã䜿çšã§ããŸãã
- åºåãååŸãããã¯ã©ã¹ã¿ãŒã® Amazon S3 ãŸã㯠HDFS ããåºåãååŸããŸããTableau ã MicroStrategy ãªã©ã®ããŒã«ã䜿çšããŠããŒã¿ãèŠèŠåããŸããAmazon EMR ã¯åŠçå®äºåŸã«èªåçã«ã¯ã©ã¹ã¿ãŒã忢ããŸãããŸãã¯ãã¯ã©ã¹ã¿ãŒããã®ãŸãŸã«ããŠããã«äœæ¥ããããšãã§ããŸãã
ãµã³ãã«ã¯ã©ã¹ã¿ãŒãèµ·åããŠã¿ãŸãããããããã¯ãªãã¯ããŠå
¥éã¬ã€ãã®ãã¥ãŒããªã¢ã«ãã芧ãã ããããã¥ãŒããªã¢ã«ã§ã¯ããµã³ãã«ããã¹ããã¡ã€ã«ã®åèªã®é »åºŠãã«ãŠã³ãããã¯ã©ã¹ã¿ãŒãäœæããŸããæ°åã§ã¯ã©ã¹ã¿ãŒã皌åããŸãã
ãµã€ã³ã¢ããã¯ç°¡åïŒç¡æç¯å²å
ã§å¯Ÿè±¡è£œåã 12 ãµæç¡æã§ã詊ãããã ããŸãã
AWS ã¢ã«ãŠã³ãäœæã®æµãã¯ãã¡ã â«
AWS ç¡æå©çšæ ã®è©³çްã¯ãã¡ã â«
å°å
¥ã«é¢ããã質åããèŠç©ããªã©ãäžæãªç¹ã¯ããæ°è»œã«æ¥æ¬æ
åœããŒã ãŸã§ãçžè«ãã ããã

