Jimey’s Life && Jimey’s World » Cloud Computing

Ubuntu10.04 编译hadoop-${version}-eclipse- plugin.jar and hadoop-0.20.3-dev-eclipse-plugin.jar

jimey — Thu, 06 May 2010 07:30:20 +0000

先是准备工作,需要java5和apache-forrest-0.8

基本上也是问题一堆,因为ubuntu9.10开始不支持java5所以装java5-sun-jdk稍微麻烦了一下

可以参考这篇文章 http://blog.csdn.net/sunrock/archive/2010/04/29/5542989.aspx

修改源改成9.04的源然后安装java5-sun-jdk

接着

安装apache-forrest-0.8

http://forrest.apache.org/mirrors.cgi For UNIX operating systems: apache-forrest-0.8.tar.gz
解压后我放在 /home/cloud/apache-forrest-0.8

直接下载来hadoop-0.20.2+228.tar.gz 解压缩即可

然后修改3个地方

1.修改$HADOOP_HOME/src/contrib/build-contrib.xml
增加一行：

2.修改 $HADOOP_HOME/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/launch/HadoopApplicationLaunchShortcut.java
注释掉原来的//import org.eclipse.jdt.internal.debug.ui.launcher.JavaApplicationLaunchShortcut;
改为import org.eclipse.jdt.debug.ui.launchConfigurations.JavaApplicationLaunchShortcut;

3.修改$HADOOP_HOME/build.xml

增加

这2行

然后ant compile

ant package 应该是没问题了

生成的eclipse plugin是在$HADOOP_HOME/build/contrib/eclipse-plugin/

或者直接去 http://hadoop-eclipse-plugin.googlecode.com/files/hadoop-0.20.3-dev-eclipse-plugin.jar 下载吧

几篇关于twitter,hadoop,Cassandra,Pig,FlockDB的slide

jimey — Thu, 22 Apr 2010 07:02:02 +0000

http://www.slideshare.net/kevinweil/nosql-at-twitter-nosql-eu-2010

http://www.slideshare.net/ryansking/scaling-twitter-with-cassandra

http://www.slideshare.net/nkallen/q-con-3770885

Big Data in Real-Time at Twitter

http://www.slideshare.net/al3x/building-distributed-systems-in-scala

主要分析了功能上需求与分散系统采用所能完成的目的的合理搭配.

个人觉得是很不错的几篇slide有助于对twitter的一个基本了解,当然是背后构架的基本了解,各个功能与服务是通过什么来实现以及达到一个怎么样的高度

具体内容各位还是看slide吧,谁看谁知道.

^^^^^^^^^^^^

再记下来本书名叫

Programming Scala (O’Reilly 2009)

云端医疗信息系统的不成熟构思

jimey — Mon, 12 Apr 2010 13:10:41 +0000

其实题目想叫着医疗信息系统云计算化或者叫医疗信息化云计算

不过 – - 都觉得无法表达我想说的意思,所以把仨题目都丢这

昨晚本来是睡觉的时候,在构思了下我之前曾经发过的一篇blog 韩国医院看病流程也许不是普遍事例

关于医疗系统的云化构思,在床上想了差不多一个小时,又来又爬起来

在医院信息系统云计算,不同于现有的OA,ERP,SQL数据库的模式.基于纯web的模式

下午在跟davidpanchina同学探讨关于这个云的具体实施想法

我才明白.医生工作站在国内没有市场，因为卫生部不认可电子病历。国内的情况是HIS+影像库，HIS应用主要是挂号，收费，药房，财务系统。

我曾经的理解是: 现在老外做云OA已经是蛮成熟了.我个人理解现在的医院信息系统都是医生桌面软件+远端DB,桌面软件开发大多都是采用vb;vc;.net之类.而云的话,就是软件处理+DB全部在云端.医生这边只用web来和云做沟通.

果然到国内去就变样了.当然目前从我观察来看,韩国应该是可以这样做

有几个问题:

1.对比国内医生部分不会使用电脑,韩国医院基本上已经实现了信息化

2. 数据安全,数据隐私

3. IC病历卡与整个系统连结

4. 医生无盘工作站的桌面 (cost可以节约一些)

5. 投入与产出,国内医院目前应该还是个卖方市场,对于IT行业的投入与倾向的喜好?

6. 各个医院数据的统一与协调,当然这个数据即使不统一,对于云数据库来说,其实是个很容易解决的问题,反正都是key-value存储,即使value短那么一截,也没什么大不了,把能对得上的对上即可

7. 如果采用private cloud 在医院内部解决数据隐私的问题的话, 与现有的集群服务器又有什么差别? 答案: 如果采用Open Source的软件的话,价格上可能会有优势

另外接着说那个影像库

1. 其实就是个存储系统,说白了,如果往云上靠,那么可以用hdfs来做,用分布式存储的模式来 vs 目前的集群服务器容错机制

^^^^^^^^^^^^^^^^

最后讨论到最后也没什么特别的结论

当然我自己个人主要是想挂靠几个理念,1.云, 2.绿色IT, 3.技术革命, 4.成本, 5.高效率

至于谁优谁劣倒是没说太多,只是想说云可以满足今后医疗信息系统的需求,当然目前来看可能国内还不太需要这个玩意,但是,相信经过一段时期发展,比如医生基本上都可以独立操作电脑,病患者可以独立操作IC卡读卡器,无人充值机,以及医院领导层对医院执行效率的提升的需求,另外是领导层对科技的倾向,软件正版化趋势(比如目前的产品采用盗版的话软件成本当然会低很多),

心中构思的未来医院是:

先说门诊部:

1. 病人进入医院,有几位引导人员对看病流程做一个介绍,并且帮助不会使用无人挂号机的病人

2. 病人带着IC病历卡在门诊前台无人机挂号(类似目前地铁IC卡无人充值机默认),目的是解决医院挂号窗口少,队伍长排队的问题

3. 门诊医生在遇到病人的时候,拿病人的IC病历卡插入读卡器,在无盘工作站上登入云,针对病人病情做出诊断,然后把诊断情况存入云端数据库,比如需要进一步化验的话,可以请病人进一步化验

4. 病人拿着IC卡到化验处插卡,比如验血,化验部抽血病人即可离开,化验部门会通知病人几点几分到专用设备前查看结果

5. 化验部门把化验结果数据全部存入云端数据库,并且通知门诊医生,门诊医生在针对化验结果上做出自己的判断,比如开处方,比如再见面,比如化验其他项目

6. 待到几点几分的时候,病人带上IC卡可以在医院专用无人读卡设备前查询化验结果和门诊医生的意见or处方,可以打印结果,等等功能

7. 如果是处方的话,病人可以打印处方然后在外面药店自己买药,或者不打印处方,在医院内部联网的处方药窗口,递卡买药即可

住院部:

1. 医生不再拿着病历一个床一个床的看,直接弄台手持设备即可,比如ipad,比如各种table的笔记本

2. 各个病人有RFID tag,或者戴手腕上的条形码或者其他快捷识别方式,住院医生只需手持装置能够读取即可

3. 住院医生可根据病人状况,通过手持装置把数据写入云数据库

…………………

当然这些都是我不成熟的想法,可能很多就现实来说先决条件也许都是错误的, 我只是想来一个Brainstorm

最后,此文内容少许有保留,如果你想继续探讨,请留下你的看法与见解.

Karmasphere Studio for Hadoop

jimey — Wed, 02 Dec 2009 06:21:42 +0000

Karmasphere Studio for Hadoop is a MapReduce development environment (IDE) based on NetBeans. It enables you to rapidly prototype MapReduce jobs without the need for a cluster, and to deploy, monitor and debug jobs on Hadoop clusters or Amazon Elastic MapReduce clusters, all from within a graphical desktop IDE.

Eclipse+Hadoop 全linux环境开发

jimey — Fri, 09 Oct 2009 09:42:58 +0000

http://trac.nchc.org.tw/cloud/wiki/waue/2009/0617

这篇文章已经叙述的很不错了

采用eclipse3.3 暂时是不会有任何问题，执行run as hadoop 那个wordcount会有错误提示

09/10/09 17:17:25 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override

properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Usage: wordcount

代码问题，可以暂时不用管

另外文章中给出的

JarFile="sample-0.1.jar"
MainFunc="Sample.WordCount"
LocalOutDir="/tmp/output"

all:help
jar:
  jar -cvf ${JarFile} -C bin/ .

run:
  hadoop jar ${JarFile} ${MainFunc} input output

clean:
  hadoop fs -rmr output

output:
  rm -rf ${LocalOutDir}
  hadoop fs -get output ${LocalOutDir}
  gedit ${LocalOutDir}/part-r-00000 & 

help:
  @echo "Usage:"
  @echo " make jar     - Build Jar File."
  @echo " make clean   - Clean up Output directory on HDFS."
  @echo " make run     - Run your MapReduce code on Hadoop."
  @echo " make output  - Download and show output file"
  @echo " make help    - Show Makefile options."
  @echo " "
  @echo "Example:"
  @echo " make jar; make run; make output; make clean"

这个makefile很不错，复制的时候注意使用tab

然后运行编译在eclipse3.3和3.4版本都很正常

目前有几个问题就是eclipse3.4版用hadoop-0.20.0-eclipse-plugin.jar 我这边出错，读不到这个插件，
用hadoop-0.19.2-eclipse-plugin.jar的时候连接不到本地的hadoop服务

还好用我最新编译的hadoop-0.20.1-eclipse-plugin.jar 可以正常使用，不过蛮搞笑的是用了这个，
在perspective添加了MapReduce的时候那个Map/Reduce Locations不会显示出来，要手动添加，
我不知道是我个人问题，还是普遍情况

目前都可以连接到我的hadoop服务上，可以在eclipse里面查看上传文件

eclipse3.3版本我是用0.20.0的eclipse-plugin

eclipse3.4版本我是用我自己编译的0.20.1的plugin  下载hadoop-0.20.1-eclipse-plugin.jar

编译hadoop-${version}-eclipse-plugin.jar and hadoop-0.20.1-eclipse-plugin.jar download

jimey — Fri, 09 Oct 2009 07:00:23 +0000

hadoop最新的0.20.1包里面已经没带eclipse的plugin了

之前用0.20.0的pluggin偶尔会出问题

所以想自编译之

从http://wiki.apache.org/hadoop/EclipsePlugIn

查阅

To build the Eclipse plug-in, you need the Hadoop source files and a working Eclipse environment (version 3.3+). When compiling Hadoop, the Eclipse plug-in will be built if it founds the Eclipse environment path in the ant property "eclipse.home". The build framework looks for this property in ${hadoop-src-root}/src/contrib/eclipse-plugin/build.properties and in $HOME/eclipse-plugin.build.properties.

A typical $HOME/eclipse-plugin.build.properties file would contain the following entry: eclipse.home=/path/to/eclipse

Then the plug-in should be built when compiling Hadoop: ant clean package (from the ${hadoop-src-root} directory), which will produce {hadoop-src-root}/build/contrib/eclipse-plugin/hadoop-${version}-eclipse-plugin.jar

To install the generated plug-in in your Eclipse environment, remove first all previous versions of the plug-in from your Eclipse environment and copy the hadoop-${version}-eclipse-plugin.jar file generated as described above in your ${eclipse.home}/plugins/ directory. When you restart Eclipse, the Map/Reduce perspective should be available.

得知只需要在 eclipse-plugin.build.properties 添加eclipse的路径即可，没找到这个eclipse-plugin.build.properties 只找到了build.properties 添加了eclipse的安装路径

然后在hadoop的根目录 ant clean package 生成 (需要loading一些classpath) 难道不能单独ant?? 在eclipse-plus下也有build.xml 用ant compile之会出现classpath错误生成60多个错误…

所以目前比较奇怪..

update:

换了个思路,不跟着那个网站的走不用那个ant clean package的命令

先在hadoop根目录下ant compile

再到src/contrib/eclipse-plugin/下

ant compile然后ant jar

这个时候会提示

BUILD FAILED
/home/hadoop/hadoop-0.20.1/src/contrib/eclipse-plugin/build.xml:69: Warning: Could not find file /home/hadoop/hadoop-0.20.1/build/hadoop-${version}-core.jar to copy.

再去src/contrib/ 编辑build-contrib.xml 添加

然后保存后

修改之前我习惯性的把hadoop-${version}目录改为hadoop再改回原来的，即/home/hadoop/hadoop/ 改为/home/hadoop/hadoop-0.20.1/

复制hadoop根目录下的hadoop-0.20.1-core.jar 到build/下

然后再去src/contrib/eclipse-plugin/下 ant jar 即可/home/hadoop/hadoop-0.20.1/build/contrib/eclipse-plugin/hadoop-0.20.1-eclipse-plugin.jar 生成

下载hadoop-0.20.1-eclipse-plugin.jar

hadoop-0.20.0

jimey — Fri, 01 May 2009 05:33:36 +0000

性能好像好了很多。

之前0.19.1版本运行完至少要1分钟

这个10秒都不用…

bin/hadoop jar hadoop-0.20.0-examples.jar wordcount dir dir-out1
09/05/01 14:28:30 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
09/05/01 14:28:30 INFO input.FileInputFormat: Total input paths to process : 1
09/05/01 14:28:31 INFO mapred.JobClient: Running job: job_local_0001
09/05/01 14:28:31 INFO input.FileInputFormat: Total input paths to process : 1
09/05/01 14:28:31 INFO mapred.MapTask: io.sort.mb = 100
09/05/01 14:28:31 INFO mapred.MapTask: data buffer = 79691776/99614720
09/05/01 14:28:31 INFO mapred.MapTask: record buffer = 262144/327680
09/05/01 14:28:32 INFO mapred.MapTask: Starting flush of map output
09/05/01 14:28:32 INFO mapred.JobClient: map 0% reduce 0%
09/05/01 14:28:32 INFO mapred.MapTask: Finished spill 0
09/05/01 14:28:32 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
09/05/01 14:28:32 INFO mapred.LocalJobRunner:
09/05/01 14:28:32 INFO mapred.TaskRunner: Task ‘attempt_local_0001_m_000000_0′ done.
09/05/01 14:28:32 INFO mapred.LocalJobRunner:
09/05/01 14:28:32 INFO mapred.Merger: Merging 1 sorted segments
09/05/01 14:28:32 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 267034 bytes
09/05/01 14:28:32 INFO mapred.LocalJobRunner:
09/05/01 14:28:33 INFO mapred.JobClient: map 100% reduce 0%
09/05/01 14:28:33 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
09/05/01 14:28:33 INFO mapred.LocalJobRunner:
09/05/01 14:28:33 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now
09/05/01 14:28:33 INFO output.FileOutputCommitter: Saved output of task ‘attempt_local_0001_r_000000_0′ to dir-out1
09/05/01 14:28:33 INFO mapred.LocalJobRunner: reduce > reduce
09/05/01 14:28:33 INFO mapred.TaskRunner: Task ‘attempt_local_0001_r_000000_0′ done.
09/05/01 14:28:34 INFO mapred.JobClient: map 100% reduce 100%
09/05/01 14:28:34 INFO mapred.JobClient: Job complete: job_local_0001
09/05/01 14:28:34 INFO mapred.JobClient: Counters: 14
09/05/01 14:28:34 INFO mapred.JobClient:   FileSystemCounters
09/05/01 14:28:34 INFO mapred.JobClient:     FILE_BYTES_READ=551968
09/05/01 14:28:34 INFO mapred.JobClient:     HDFS_BYTES_READ=1383924
09/05/01 14:28:34 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=569086
09/05/01 14:28:34 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=515816
09/05/01 14:28:34 INFO mapred.JobClient:   Map-Reduce Framework
09/05/01 14:28:34 INFO mapred.JobClient:     Reduce input groups=0
09/05/01 14:28:34 INFO mapred.JobClient:     Combine output records=18042
09/05/01 14:28:34 INFO mapred.JobClient:     Map input records=12761
09/05/01 14:28:34 INFO mapred.JobClient:     Reduce shuffle bytes=0
09/05/01 14:28:34 INFO mapred.JobClient:     Reduce output records=0
09/05/01 14:28:34 INFO mapred.JobClient:     Spilled Records=36084
09/05/01 14:28:34 INFO mapred.JobClient:     Map output bytes=1086784
09/05/01 14:28:34 INFO mapred.JobClient:     Combine input records=109855
09/05/01 14:28:34 INFO mapred.JobClient:     Map output records=109855
09/05/01 14:28:34 INFO mapred.JobClient:     Reduce input records=18042

Hadoop+MySQL结合

jimey — Tue, 28 Apr 2009 12:09:17 +0000

……

hadoop mail list 收到的

给没订阅的童鞋门参考

author : Ankur Goel [ankur.goel#corp.aol.com]

………………

hello hadoop users,

Recently I had a chance to lead a team building a log-processing system that uses Hadoop and MySQL. The system’s goal was to process the incoming information as quickly as possible (real time or near real time), and make it available for querying in MySQL. I thought it would be good to share the experience and the challenges with the community. Couldn’t think of a better place than these mailing lists as I am not much of a blogger

The information flow in the system looks something like

[Apache-Servers] -> [Hadoop] -> [MySQL-shards] -> [Query-Tools]

Transferring from Apache-Servers to Hadoop was quite easy as we just had to organize the data in timely buckets (directories). Once that was running smooth we had to make sure that map-reduce jobs are fired at regular intervals and they pick up the right data. The jobs would then process/aggregate the date and dump the info into MySQL shards from the reducers [we have our own DB partioning set up]. This is where we hit major bottlenecks [any surprises? ]

The table engine used was InnoDB as there was a need for fast replication and writes but only moderate reads (should eventually support high read rates). The data would take up quite a while to load completely far away from being near-real time. And so our optimization journey begin.

1. We tried to optimize/tune InnoDB parameters like increasing the buffer pool size to 75 % of available RAM. This helped but only till the time DBs were lightly loaded i.e. innoDB had sufficient buffer pool to host the data and indexes.

2. We also realized that InnoDB has considerable locking overhead because of which write concurrency is really bad when you have a large number of concurrent threads doing writes. The default thread concurrency for us was set to no_of_cpu * 2 = 8 which is what the official documentation advises as the optimal limit. So we limited the number of reduce tasks and consequently the number of concurrent writes and boy the performance improved 4x. We were almost there

3. Next thing we tried is the standard DB optimzation techniques like de-normalizing the schema and dropping constraints. This gave only a minor performance improvement, nothing earth shattering. Note that we were already caching connections in reducers to each MySQL shard and partionining logic was embedded into reducers.

4. Falling still short of our performance objectives, we finally we decided to get rid of JDBC writes from reducers and work on an alternative that uses MySQLs LOAD utility.

- The processing would partition the data into MySQL shard specific files resident in HDFS.

- A script would then spawn processes via ssh on different physical machines to download this data.

- Each spawned process just downloads the data for the shard it should upload to.

- All the processes then start uploading data in parallel into their respective MySQL shards using LOAD DATA infile.

This proved to be the fastest approach, even in the wake of increasing data loads. The enitre processing/loading would complete in less than 6 min. The system has been holding up quite well so far, even though we’ve had to limit the number of days for which we keep the data or else the MySQLs get overwhelmed.

Hope this is helpful to people.

Regards

-Ankur

[ZT]Writing your own real-time search engine and Adwords service

jimey — Thu, 16 Apr 2009 13:54:07 +0000

In some cases this technique is not applicable. For example, imagine the case of eCommerce applications. When you submit a bid on an auction, you expect to see it immediately popping up in the search results.This calls for a rather different architecture then the one typically used for Internet applications. The key is how fast we can put new data into the index server. Sounds simple, right? Well, most index servers are highly optimized for fast read, however they tend to be quite heavy on write operations.

[via]

无意间找到的，好像是我搜索push的时候找到的，整个博客的内容就是我想要的东西，剩下这几天有得看了

几个分享和想法

jimey — Thu, 16 Apr 2009 04:36:34 +0000

1.push

2.mobile search

http://www.readwriteweb.com/archives/new_mishmash_of_a_mobile_search_engine_getfugu_pre.php

3 one login

http://www.readwriteweb.com/archives/myonelogin_single_sign-on_for_the_cloud.php

方案1暂时在调查中还不能放出更多detail 但是以这个单词为主题的内容也就那么一些

方案2 很不错的想法，基于LBS 的应用，与搜索引擎结合很不错，只是比以前的想法多了一个图片拍照与识别之后的搜索，更加便捷一些

当然，你可以加语音识别，等等等等一系列的识别上去，目的都是为了在有限的时间内，做更多的事情

方案3 概念还真的跟银行类似，把网站id 密码存银行，然后要上哪个网站，先进银行就好了。但是万一这银行倒闭怎么办，长期不输入帐号密码，再叫我去输入，我肯定会忘记的。。所以该银行每个月服务费3美元，以维持他不倒闭

4 ubiquitous computing + cloud computing 会成什么样？

昨天一直在想那个关于搜索idea的东西，如果ubiquitous computing + cloud computing 又会是如何？

sensor 满大街丢，手机汽车人交通灯树垃圾箱等等一切都装上这个sensor的话又会如何？

很多人在想，我为何要在我身上装一个只能发射消息，转发消息而不能给我带来任何好处的sensor?? 那就好比为何我要天天在twitter上发消息又不能给我带来任何有用的收获一样的问题。。

因为如果每个人都为其他人做一些贡献，做一些共享的话，最终自己也会是其中一受益人

就好比每天在twitter发一些看似无趣，无意义的内容，也许会对某些人有所帮助，即使无帮助，也不会带给太多人以影响，只是对那些follower们有影响，在搜索过滤系统的作用下，不会对更多人有影响

再说回这个ubiquitous computing + cloud computing 以sensor收集传输信息与手持装置互动。再加上强大的cloud computing作为强力计算源泉，analysis sensor所提供的信息，utilize cloud computing 满足各手持装置(手机，pda，车载系统，移动笔记本，等等)的请求创造强大和谐社会的目标是可以提前实现的

通过sensor来分析人流密度，交通密度，对树木进行园林管理，对垃圾箱清理时间进行安排

对于人流密度，又可以更加和谐的来用到商业上，比如营销，发广告传单的时候 lol

而交通密度，其实不仅仅是交通密度的一个指示，更多的扩展来讲，可以进行无人驾驶，通过sensor来感知对方车辆与自己的距离，速度，以及前进目的等等信息，利用gps 街边hot spot 接收车载计算系统来计算最优化的前进路线，这又是需要多么牛X的AI在里面吧

说来说去，其实都是还在为了人类能够在自己短短的生命里面做更多的事情，就好比无人驾驶，好比那些最优化设计，都是为了更快，更好，不用更多人的完成事情

说来说去，其实我们是在造人运动，这sensor就相当于人类的眼睛鼻子手测速器等等而车载电脑or cloud computing 就相当于大脑，这AI其实就是在用科学的方法寻找一个最终的选择

人总是会因为信息过量而会崩溃，比如每天我在看我的google reader的时候看着左上角那上百条更新的，十分的想崩溃

一直在试想，如果有机器能够替我筛选，选我一定会有收获，会有理解，可以满足我的好奇心，可以把我不认识的单词都翻译了，对于那些我完全没兴趣直接标记为已读，对于那些我今天暂时不想看，也许明天会看的暂时屏蔽掉。让我不要花费半个小时甚至一个小时时间来看这上百个更新只要最少的时间把该看的都看掉，那生活就美好了

还是那么一句话，节约了我的时间，其实是为了能够让我在我有限的时间里面干更多的事情

也许有人会讲：难道你写博客不是在浪费自己的时间？这个其实跟上面那段，我干嘛要带个sensor满街跑，我干嘛要在twitter上无聊发消息一样的问题。答案其实也是，在你分享你的想法，你的收获的时候，最终大家都这样做的时候，这样其实你是在分享给你自己

windows环境下eclipse调试远端ubuntu环境的hadoop

jimey — Thu, 26 Mar 2009 12:32:21 +0000

基本上这个link说的很明白

强烈推荐

在run dialog中设置输入参数这项，我有点sb了

以为是真实环境目录，后来出错才发现应该是hdfs中的虚拟目录……

在run方法中加入下面2句

conf.set("hadoop.job.ugi", "hadoop,hadoop"); //设置hadoop server用户名和密码

conf.set("mapred.system.dir", "/home/hadoop/HadoopInstall/tmp/mapred/system/"); //指定系统路径

参照之前的 yahoo的教程所述

click on the "Advanced" tab. There are two settings here which must be changed.

Scroll down to hadoop.job.ugi. It contains your current Windows login credentials. Highlight the first comma-separated value in this list (your username) and replace it with hadoop-user.

Next, scroll further down to mapred.system.dir. Erase the current value and set it to /hadoop/mapred/system.

在eclipse插件上修改这个参数。。但是完全不在我系统上显示，这个hadoop.job.ugi……所以总是无法连接

当然目前由于这个无法出现，在eclipse直接上传文件的功能暂时就无效了，当然，这个没所谓，还有其他办法，svn put.ftp等等等

重要的是，终于可以在windows 环境下eclipse中debug了。不用再把code put上去再运行了……

Linux server clustered HDFS: access from Windows eclipse Java application

jimey — Wed, 25 Mar 2009 13:17:53 +0000

貌似有人也有一样的问题，找不到hadoop.job.ugi

不过我也找不到。。搞了3个小时实在是奇怪

还是用svn put上去好了。。

正常的配置是

You will now be asked to fill in a number of parameters identifying the server. To connect to the VMware image, the values are:

Location name: (Any descriptive name you want; e.g., "VMware server")Map/Reduce Master Host: (The IP address printed at startup)Map/Reduce Master Port: 9001DFS Master Port: 9000User name: hadoop-user

Next, click on the "Advanced" tab. There are two settings here which must be changed.

Scroll down to hadoop.job.ugi. It contains your current Windows login credentials. Highlight the first comma-separated value in this list (your username) and replace it with hadoop-user.

Next, scroll further down to mapred.system.dir. Erase the current value and set it to /hadoop/mapred/system.

When you are done, click "Finish." Your server will now appear in the Map/Reduce Locations panel. If you look in the Project Explorer (upper-left corner of Eclipse), you will see that the MapReduce plugin has added the ability to browse HDFS. Click the [+] buttons to expand the directory tree to see any files already there. If you inserted files into HDFS yourself, they will be visible in this tree.

[via]

但是目前比较奇怪找不到这个hadoop.job.ugi无法显示。。

by Alberto Forcén Aug 05, 2008; 06:16pm :: Rate this Message: – Use ratings to moderate (?)

Reply | Reply to Author | Print | View Threaded | Show Only this Message

Hi all.

I’m running a clustering HDFS on linux and I need to access files (I/O) from eclipse Java application running on Windows. It seems simple, but is it possible?

I have write code using API but I have a problem: when code invokes DistributedFileSystem.initialize() method I receive an exception: java.net.SocketTimeoutException

[code]

String ipStr = "192.168.75.191";

String portStr = "9000";

String uriStr = "http://" + ipStr + ":" + portStr;

Configuration conf = new Configuration();

conf.set("hadoop.job.ugi", "user,group"); // Usuario y grupos a los que pertenece

DistributedFileSystem dfs = new DistributedFileSystem();

dfs.initialize(new URI(uriStr), conf);

[/code]

[trace]

Exception in thread "main" java.net.SocketTimeoutException: timed out waiting for rpc response

at org.apache.hadoop.ipc.Client.call(Client.java:559)

at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)

at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)

at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)

at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)

at org.apache.hadoop.dfs.DFSClient.(DFSClient.java:178)

at org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)

at examples.HadoopDFS.main(HadoopDFS.java:153)

[/trace]

______________________________________________

Enviado desde Correo Yahoo! La bandeja de entrada más inteligente.

Re: Linux server clustered HDFS: access from Windows eclipse Java application

by Qin Gao-2 Aug 05, 2008; 10:13pm :: Rate this Message: – Use ratings to moderate (?)

Reply | Reply to Author | Print | View Threaded | Show Only this Message

I think IBM has a plugin that can access HDFS, I don’t know whether it

contains source code, but maybe it helps.

www.alphaworks.*ibm*.com/tech/mapreducetools

On Tue, Aug 5, 2008 at 5:16 AM, Alberto Forcén <otrebla81@…> wrote:

> Hi all.

>

> I’m running a clustering HDFS on linux and I need to access files (I/O)

> from eclipse Java application running on Windows. It seems simple, but is it

> possible?

> I have write code using API but I have a problem: when code invokes

> DistributedFileSystem.initialize() method I receive an exception:

> java.net.SocketTimeoutException

> [code]

> String ipStr = "192.168.75.191";

> String portStr = "9000";

> String uriStr = "http://" + ipStr + ":" + portStr;

> Configuration conf = new Configuration();

> conf.set("hadoop.job.ugi", "user,group"); // Usuario y grupos a los que

> pertenece

> DistributedFileSystem dfs = new DistributedFileSystem();

> dfs.initialize(new URI(uriStr), conf);

> [/code]

> [trace]

> Exception in thread "main" java.net.SocketTimeoutException: timed out

> waiting for rpc response

> at org.apache.hadoop.ipc.Client.call(Client.java:559)

> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)

> at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)

> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)

> at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)

> at org.apache.hadoop.dfs.DFSClient.(DFSClient.java:178)

> at

> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)

> at examples.HadoopDFS.main(HadoopDFS.java:153)

> [/trace]

> ______________________________________________

> Enviado desde Correo Yahoo! La bandeja de entrada más inteligente.

… [show rest of quote]

Confusing NameNodeFailover page in Hadoop Wiki

by Konstantin Shvachko Aug 06, 2008; 03:42am :: Rate this Message: – Use ratings to moderate (?)

Reply | Reply to Author | Print | View Threaded | Show Only this Message

I was wondering around Hadoop wiki and found this page dedicated to name-node failover.

http://wiki.apache.org/hadoop/NameNodeFailover

I think it is confusing, contradicts other documentation on the subject and contains incorrect facts. See

http://hadoop.apache.org/core/docs/current/hdfs_user_guide.html#Secondary+Namenode

http://wiki.apache.org/hadoop/FAQ#7

Besides it contains some kind of discussion.

It is not that I am against discussions, lets have them on this list.

But I was trying to understand were all the confusion about secondary-node issues comes from lately…

Imho we either need to correct it or remove.

Thanks,

–Konstantin

Re: Linux server clustered HDFS: access from Windows eclipse Java application

by Michael Bieniosek Aug 06, 2008; 03:56am :: Rate this Message: – Use ratings to moderate (?)

Reply | Reply to Author | Print | View Threaded | Show Only this Message

Are you sure both ends are running the same version of hadoop?

On 8/5/08 2:16 AM, "Alberto Forcén" <otrebla81@…> wrote:

Hi all.

I’m running a clustering HDFS on linux and I need to access files (I/O) from eclipse Java application running on Windows. It seems simple, but is it possible?

I have write code using API but I have a problem: when code invokes DistributedFileSystem.initialize() method I receive an exception: java.net.SocketTimeoutException

[code]

String ipStr = "192.168.75.191";

String portStr = "9000";

String uriStr = "http://" + ipStr + ":" + portStr;

Configuration conf = new Configuration();

conf.set("hadoop.job.ugi", "user,group"); // Usuario y grupos a los que pertenece

DistributedFileSystem dfs = new DistributedFileSystem();

dfs.initialize(new URI(uriStr), conf);

[/code]

[trace]

Exception in thread "main" java.net.SocketTimeoutException: timed out waiting for rpc response

at org.apache.hadoop.ipc.Client.call(Client.java:559)

at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)

at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)

at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)

at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)

at org.apache.hadoop.dfs.DFSClient.(DFSClient.java:178)

at org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)

at examples.HadoopDFS.main(HadoopDFS.java:153)

[/trace]

______________________________________________

Enviado desde Correo Yahoo! La bandeja de entrada más inteligente.

Re: Confusing NameNodeFailover page in Hadoop Wiki

by Doug Cutting-4 Aug 07, 2008; 01:39am :: Rate this Message: – Use ratings to moderate (?)

Reply | Reply to Author | Print | View Threaded | Show Only this Message

Konstantin Shvachko wrote:

> Imho we either need to correct it or remove.

Doug

Re: Confusing NameNodeFailover page in Hadoop Wiki

by Steve Loughran Aug 07, 2008; 07:34pm :: Rate this Message: – Use ratings to moderate (?)

Reply | Reply to Author | Print | View Threaded | Show Only this Message

Doug Cutting wrote:

> Konstantin Shvachko wrote:

>> Imho we either need to correct it or remove.

> +1

> Doug

I added some pages there on namenode/jobtracker, etc, linking to the

faiover doc, which I didnt compare to the svn docs to see what was

correct. Perhaps the failover page could be set up to say "you can do

some things here" and point to the full docs at SVN or the hadoop site

–

Steve Loughran http://www.1060.org/blogxter/publish/5

Author: Ant in Action http://antbook.org/

基于Eclipse的Hadoop应用开发环境的配置

jimey — Wed, 25 Mar 2009 09:48:03 +0000

最新版的Hadoop 0.19.1中

hadoop-0.19.1\contrib\eclipse-plugin

有新版的hadoop-0.19.1-eclipse-plugin 插件

只可惜暂时不支持最新版的eclipse

所以无奈只能http://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/europa/winter/eclipse-java-europa-winter-win32.zip

选择europa版本的eclipse进行配置

Big Data: Technologies and Techniques for Large-Scale Data

jimey — Mon, 23 Mar 2009 15:39:01 +0000

[via]

Zotero 1.5 Beta Released: Join Us In The Clouds

jimey — Wed, 25 Feb 2009 05:55:59 +0000

Zotero 1.5 Beta Released: Join Us In The Clouds

Posted February 23rd, 2009 by Trevor

We are excited to announce two major releases. This morning the Zotero team launched both Zotero 1.5 and the first release of Zotero’s web application. Watch the screencast below for details on 1.5 and see the list below for a brief list of Zotero’s new features.

Automatic synchronization of collections across multiple computers. For example, sync your PC at work with your Mac laptop and your Linux desktop at home.

Free automatic backup of your library data at Zotero.org.

Automatic synchronization of your attachment files to a server of your choice (e.g. iDisk, Jungle Disk, or university-provided web storage).

Recover recently deleted items with Zotero’s trash can.

Rich-text notes

New style manager allowing you to add and delete CSLs and legacy style formats

Automatic detection of PDF metadata (author, title, etc.)

Automatic detection and support for proxy servers

[via]

Cloud Computing defined by Berkeley RAD Labs

jimey — Sat, 14 Feb 2009 07:11:19 +0000

Executive summary:

Cloud Computing, the long-held dream of computing as a utility, has the potential to transform a large part of the IT industry, making software even more attractive as a service and shaping the way IT hardware is designed and purchased. Developers with innovative ideas for new Internet services no longer require the large capital outlays in hardware to deploy their service or the human expense to operate it. They need not be concerned about over-provisioning for a service whose popularity does not meet their predictions, thus wasting costly resources, or under-provisioning for one that becomes wildly popular, thus missing potential customers and revenue. Moreover, companies with large batch-oriented tasks can get results as quickly as their programs can scale, since using 1000 servers for one hour costs no more than using one server for 1000 hours. This elasticity of resources, without paying a premium for large scale, is unprecedented in the history of IT. Cloud Computing refers to both the applications delivered as services over the Internet and the hardware and systems software in the datacenters that provide those services. The services themselves have long been referred to as Software as a Service (SaaS). The datacenter hardware and software is what we will call a Cloud. When a Cloud is made available in a pay-as-you-go manner to the general public, we call it a Public Cloud; the service being sold is Utility Computing. We use the term Private Cloud to refer to internal datacenters of a business or other organization, not made available to the general public. Thus, Cloud Computing is the sum of SaaS and Utility Computing, but does not include Private Clouds. People can be users or providers of SaaS, or users or providers of Utility Computing. We focus on SaaS Providers (Cloud Users) and Cloud Providers, which have received less attention than SaaS Users. From a hardware point of view, three aspects are new in Cloud Computing:

The illusion of infinite computing resources available on demand, thereby eliminating the need for Cloud Computing users to plan far ahead for provisioning.
The elimination of an up-front commitment by Cloud users, thereby allowing companies to start small and increase hardware resources only when there is an increase in their needs.
The ability to pay for use of computing resources on a short-term basis as needed (e.g., processors by the hour and storage by the day) and release them as needed, thereby rewarding conservation by letting machines and storage go when they are no longer useful.

We argue that the construction and operation of extremely large-scale, commodity-computer datacenters at low-cost locations was the key necessary enabler of Cloud Computing, for they uncovered the factors of 5 to 7 decrease in cost of electricity, network bandwidth, operations, software, and hardware available at these very large economies of scale. These factors, combined with statistical multiplexing to increase utilization compared a private cloud, meant that cloud computing could offer services below the costs of a medium-sized datacenter and yet still make a good profit. Any application needs a model of computation, a model of storage, and a model of communication. The statistical multiplexing necessary to achieve elasticity and the illusion of infinite capacity requires each of these resources to be virtualized to hide the implementation of how they are multiplexed and shared. Our view is that different utility computing offerings will be distinguished based on the level of abstraction presented to the programmer and the level of management of the resources.

Amazon EC2 is at one end of the spectrum. An EC2 instance looks much like physical hardware, and users can control nearly the entire software stack, from the kernel upwards. This low level makes it inherently difficult for Amazon to offer automatic scalability and failover, because the semantics associated with replication and other state management issues are highly application-dependent. At the other extreme of the spectrum are application domain-specific platforms such as Google AppEngine. AppEngine is targeted exclusively at traditional web applications, enforcing an application structure of clean separation between a stateless computation tier and a stateful storage tier. AppEngine’s impressive automatic scaling and high-availability mechanisms, and the proprietary MegaStore data storage available to AppEngine applications, all rely on these constraints. Applications for Microsoft’s Azure are written using the .NET libraries, and compiled to the Common Language Runtime, a language-independent managed environment. Thus, Azure is intermediate between application frameworks like AppEngine and hardware virtual machines like EC2. When is Utility Computing preferable to running a Private Cloud? A first case is when demand for a service varies with time. Provisioning a data center for the peak load it must sustain a few days per month leads to underutilization at other times, for example. Instead, Cloud Computing lets an organization pay by the hour for computing resources, potentially leading to cost savings even if the hourly rate to rent a machine from a cloud provider is higher than the rate to own one. A second case is when demand is unknown in advance. For example, a web startup will need to support a spike in demand when it becomes popular, followed potentially by a reduction once some of the visitors turn away. Finally, organizations that perform batch analytics can use the "cost associativity" of cloud computing to finish computations faster: using 1000 EC2 machines for 1 hour costs the same as using 1 machine for 1000 hours. For the first case of a web business with varying demand over time and revenue proportional to user hours, we have captured the tradeoff in the equation below.

The left-hand side multiplies the net revenue per user-hour by the number of user-hours, giving the expected profit from using Cloud Computing. The right-hand side performs the same calculation for a fixed-capacity datacenter by factoring in the average utilization, including nonpeak workloads, of the datacenter. Whichever side is greater represents the opportunity for higher profit.

The table below previews our ranked list of critical obstacles to growth of Cloud Computing; the full discussion is in Section 7 of our paper. The first three concern adoption, the next five affect growth, and the last two are policy and business obstacles. Each obstacle is paired with an opportunity, ranging from product development to research projects, which can overcome that obstacle.

We predict Cloud Computing will grow, so developers should take it into account. All levels should aim at horizontal scalability of virtual machines over the efficiency on a single VM. In addition:

Applications Software needs to both scale down rapidly as well as scale up, which is a new requirement. Such software also needs a pay-for-use licensing model to match needs of Cloud Computing.
Infrastructure Software needs to be aware that it is no longer running on bare metal but on VMs. Moreover, it needs to have billing built in from the beginning.
Hardware Systems should be designed at the scale of a container (at least a dozen racks), which will be is the minimum purchase size. Cost of operation will match performance and cost of purchase in importance, rewarding energy proportionality such as by putting idle portions of the memory, disk, and network into low power mode. Processors should work well with VMs, flash memory should be added to the memory hierarchy, and LAN switches and WAN routers must improve in bandwidth and cost.

Table: Quick Preview of Top 10 Obstacles to and Opportunities for Growth of Cloud Computing.

Obstacle
Opportunity

1
Availability of Service
Use Multiple Cloud Providers; Use Elasticity to Prevent DDOS

2
Data Lock-In
Standardize APIs; Compatible SW to enable Surge Computing

3
Data Confidentiality and Auditability
Deploy Encryption, VLANs, Firewalls; Geographical Data Storage

4
Data Transfer Bottlenecks
FedExing Disks; Data Backup/Archival; Higher BW Switches

5
Performance Unpredictability
Improved VM Support; Flash Memory; Gang Schedule VMs

6
Scalable Storage
Invent Scalable Store

7
Bugs in Large Distributed Systems
Invent Debugger that relies on Distributed VMs

8
Scaling Quickly
Invent Auto-Scaler that relies on ML; Snapshots for Conservation

9
Reputation Fate Sharing
Offer reputation-guarding services like those for email

10
Software Licensing
Pay-for-use licenses; Bulk use sales

[via]

How Businesses Can Use P2P

jimey — Sat, 07 Feb 2009 03:43:00 +0000

Almost every description of P2P in the context of business infrastructure starts something like this: "P2P is notorious for…" This comes from many years of people associating P2P with illegal downloading, to the point that the terms are now almost synonymous. Such an association is inherently unfair, however, because no one equates TCP/IP and crime, despite the fact that TCP/IP is the protocol of choice for many cyber-criminals.

Rather than resorting to out-dated and inaccurate definitions, let’s start from scratch and consider the following: what is P2P, really? What is it good for? How can we use it to save and earn money?

What’s in a Name?

P2P stands for "peer to peer." Put simply, it’s one method of establishing communication between parties. Uploading information to Google Docs to share it with colleagues is not P2P, but sending the same information as an attachment to email is, despite the fact that mail servers are involved. In this context, P2P doesn’t mean "serverless communication" so much as "communication that is perceived to be serverless." Like email, instant messaging (IM) is considered P2P technology because even though servers are used quite extensively, there is no explicit act of uploading data to an intermediate location. With both email and IM, servers are used behind the scenes, so to speak.

Thus, "P2P" is as much a social term as a technical one. It connotes a grid or cloud of devices that are more or less equal, rather than a constellation of star-like servers with clusters of clients surrounding them. But from the purely technical point of view, there is a distinct difference between true P2P (in which data is not relayed through a server) and perceived P2P (in which data is relayed through a server, but we don’t see it happen).

P2P and the Cloud

As currently implemented, "cloud computing" is really just a new name for old-school client-server computing, except that the servers pretend to be redundant. Clients have little or no actual control over this redundancy and cannot even verify its existence. When a cloud-based service tells you that your data is stored securely, you have no choice but to trust it… or not.

Compare that with a P2P file-sharing network. On a P2P network, every peer can tell who has complete or partial copies of a given file, what percentage of the file is stored locally, as well as many other details. Doesn’t that seem like a better example of cloud storage? If not, then what is?

Even better, with this set-up you can easily control the level of redundancy: just add another client instance, have it share the same file, and you’ve increased your redundancy by one. However, you can’t reduce redundancy beyond the peers under your influence. If all peers don’t agree to remove a given file, no one can. There have been many ideas about implementing a kind of "delete button" for the web, but the closer we move towards cloud computing, the less likely such a scenario becomes.

Cloud services are chosen for their convenience (being accessible from everywhere with simple tools) and reliability (with redundant storage in stable data centers). P2P technologies increase both factors: they increase convenience because there is no uploading or downloading to and from the cloud, and they provide directly controllable redundancy and, thus, cost control.

In most cases, more reliability means higher prices, and not all data deserves the same level of service. With P2P platforms behind cloud services, developers could implement applications that allow multiple storage and processing schemes without much hassle. This is not always good for service providers, because flexible cost control means that customers can scale up and down freely as business and economic conditions demand. But for the industry as a whole, it is definitely a good thing because it stiffens competition and enables customers to better survive.

What is perhaps more significant about introducing P2P technology into cloud computing is that the P2P cloud would truly be a cloud, not just a 15-year-old client-server technology with a new sticker. If you are inviting us into the cloud, then let us truly be a part of it, instead of remaining a client that we can neither see nor control.

P2P as a Social Tool

The traditional Internet (Web 1.0) was built mostly like a television network. Websites were controlled by a small group of content producers (editors, media teams, individual owners, etc.), and millions of users were consumers of that content. The barrier to entry was far lower than it was in the television industry, but it was still mainly a one-way road from the website to the user. The Internet’s infrastructure reflected this: data centers, thick pipes that connected them, and subscriber lines that could download much faster than upload.

Then Web 2.0 came along, and everything started to change. Today, most popular sites were created by ordinary users who care less about owning content than about sharing it with others. Developers created ecosystems and gave users tools to track each other and exchange content, and sites became service providers instead of content sources.

What happened to the Internet’s infrastructure, then? Nothing. Imagine YouTube going out of business: the entire infrastructure for that video would disappear in a day, but the videos themselves would still exist, scattered among computers of individual users, stuck on cell phones, caught in caches, etc.

Now take that one step further: instead of YouTube, we have a P2P network, full of videos and convenient tools to watch and upload them. Nobody can close this network or put it out of business. Sufficiently large P2P networks are invincible; if you don’t believe that, ask the MPAA and RIAA.

Web 2.0 is P2P (in a social sense), done with Web 1.0 tools and old infrastructure. To unlock a box with a whole set of new services, we need to upgrade the infrastructure. Just as Gopher was replaced by WWW, and UUCP was replaced by SMTP, the current star-shaped web infrastructure will be replaced by a mesh-shaped cloud network. Data centers would still exist, but instead of providing bandwidth and servers, they would provide reliability and accessibility. (And a system of measurement would need to be established for both.)

We can call this a "social Internet infrastructure": an infrastructure that reflects new social behavior, that allows anyone to connect and share content with anyone else, while still enjoying sufficient privacy and security. It’s not so much a revolution as an evolution: another step in a process that has been occuring for some time already. It happened to the telephone system — does anyone remember having to call a phone station to be put through to another person? I’ve only read about it in books. It will happen to the Internet eventually, too.

So, Show Us the Money

What is P2P good for, then? To answer that, let’s first look at the potential benefits of an "ideal" P2P implementation, benefits that a business could use for competitive advantage or to implement an entirely new service.

The most talked-about advantage of P2P is its ability to cut bandwidth bills. Imagine how much money companies like Dell, Logitech and Microsoft spend on downloads of products and services? Logitech’s generic mouse driver is over 30 MB in size, and the number of its mice being used around the world is countless. A Microsoft Windows service pack can be as big as 300 MB. Now think of how many computers run Windows? (Remember that Apple has less than 25% market share, if that helps.) There are also media distribution companies, web accelerators, distributed back-ups, the list goes on. And every one of these companies could dramatically reduce its bandwidth bill by using P2P. Properly implementing P2P content distribution isn’t easy; but when our industry had (almost) more money than it could spend, nobody cared to try it. Today, money isn’t just an issue; it’s the issue.

Look at Skype. For it, implementing P2P was not optional but mandatory. Creating a free multimedia service for millions of people without spending tens of millions of dollars on "free" infrastructure wasn’t (and still isn’t) possible. With P2P, Skype was able to provide free phone access without actually subsidizing users (this is not 100% accurate, but accurate enough for this example). All the money it collects from paid users is profit, and yet its creators failed with Joost, not because no ones wants free television, but because Joost had a different infrastructure. People were not into sharing TV as much as they were into simply talking to each other. And then came YouTube, which was funded first by venture capital and then by Google.

Another benefit of P2P is that it requires zero configuration. Skype is probably not the best IP phone around, nor was it the first; but you don’t have to be a telecommunications engineer to use it. You download the installer, run it, register yourself as a user, and off you go, from nothing to brilliant conversation in a few minutes.

As with content distribution, implementing a P2P network that requires no configuration isn’t an easy task, but it dramatically reduces the number of users who drop off from being intimidated by the technology or feeling they lack the necessary skill. For many services, this is the difference between 100,000 users and 10 million users, or between going out of business as soon as venture money dries up and being profitable within a year.

Zero-configuration P2P has to do with more than just P2P, though. It also implies being able to fully network with zero configuration: the ability to connect any device anywhere using any available connection. Unattended sensors, medical IT devices, military computers, none of these should require in-field configuration. The people who use them generally don’t have time to read instruction manuals. They should be able to open the box, insert batteries, and have a workable device within seconds. This is what zero configuration gives users: a choice, not just between high and low profitability, but between life and death.

Even in the case of lower-profile applications, zero-configuration P2P can cut deployment costs tremendously — and well-implemented P2P platforms could reduce those costs to almost nothing. On such a platform, for example, setting up a new message-processing server for a financial system would be as easy as opening the box, throwing the server on the rack, plugging in the ethernet and power cables, and nothing else.

Ubiquitous connectivity simplifies development costs. Message-passing platforms wouldn’t need to account for different types of hosts, relays, connectivity fall-backs, and so on. You would simply confirm that the peer is up and then send it a message. Done. Think of how many networked hosts out there are actually "gateways" between email and text messages, between Internet and Intranets, between X and Y technologies. When one networked device can securely connect with any other, many problems simply evaporate. Before TCP/IP took the world by storm, gazillions of networking technologies existed. No one remembers their names, not because they were inadequate, but because TCP/IP was everywhere, and it’s much easier to speak the common language than to teach others a "better" one.

All of these factors are hopefully pushing cloud networking technologies towards commoditization, which would expand markets, decrease infrastructure costs, and allow companies to deliver better products more cheaply.

[via]

Cisco in The Cloud: Application Extension Platform (AXP)

jimey — Wed, 21 Jan 2009 10:57:05 +0000

One doesn’t tend to associate Cisco with Web 2.0, social media or cloud computing. Cisco equals routers, the plumbing of the web. In other words, vital and hugely profitable; but not the ‘new, new thing’ of the Internet. It turns out we have written before about how Cisco is using social media to reach enterprise clients and playing in the enterprise social networking space. Now Cisco wants to position its Application Extension Platform (AXP) in the cloud platform space and is encouraging developers with $100,000 in prizes for the most innovative applications.

The product AXP is not new. Cisco launched it about 4 years ago. But it is now ramping up the marketing. Read on for a rundown of AXP’s positioning in the market.

The Networking Problems In Branch Offices

On the consumer Internet, we connect directly to big server farms running our favorite services. In the Enterprise world, most users connect via a branch network. In this more complicated world, there are three concerns that are not well addressed by the platforms that emerged in the Web 2.0 era:

Security at the branch level. Security at the data center or server farm may be great, but the branch is often a back-door threat.
Performance. If a hundred people are working in a branch, dealing with some network traffic locally would improve performance.
Survivability. Losing connectivity to the cloud is still a significant issue. If it happens to a single consumer, you shrug it off and say "Oh well." If it stops a bank branch from functioning, well, that’s another story.

Cisco AXP aims to address these challenges in branch networks. The $100,000 contest is part of this, as it asks developers to think of what apps can run at the branch level within a Cisco network router (aka "the box") – which is really a Linux server with the network services exposed via an API. February 27th is the final deadline for submissions, if you’re a developer interested in this challenge. Finalists will be announced in May and the winners will be announced in August/September.

For more on the technology, current usage cases, and competition rules see Cisco’s presentation on Slideshare.

Cisco Media Briefing Developer Contest V4.3
View SlideShare presentation or Upload your own.

[via]

软件算法可视化大型数据集

jimey — Wed, 21 Jan 2009 06:48:23 +0000

科学家比以前拥有更多的可用数据，多到他们无力恰当检验。现在一个新的算法可以让他们更简单，也更廉价的可视化大型数据集(data set)。基于算法的软件可以运行在个人电脑上，只需2G内存。该项目由加州戴维斯分校计算机科学研究生Attila Gyulassy领导，作为他的博士论文主题。目前超级计算机能模拟如洋流和燃烧之类的物理现象，但产生的大量数据几乎不可能直接使用。Gyulassy的教授Bernd Hamann称，没有可视化工具的帮助，是无法理解数据的意义的。在笔记本上，Gyulassy以模拟两种液体融合去测试算法——一个在三维网格上超过 10亿个点的数据集。软件在24小时内分析数据，并勾画出了数秒内的现象。

无关博文

2009-11-30 -- Last.fm背后的技术 (2)
2008-09-26 -- 手机里面的乱拍的pp (1)
2008-11-01 -- 终于食了下人间烟火 (2)
2006-01-30 -- 半夜回家 (4)
2009-11-30 -- 2009.11.27 汉城首晚明洞世宗路清溪川 (6)
2009-01-03 -- 回到了一周没回的家 (8)
2009-12-21 -- 看完Avarar digital 3D了 (2)
2008-12-19 -- 今天又高调了 (0)
2009-11-13 -- bt软件uTorrent支持伊甸园bt下载站资源“全自动”下载教程 (0)
2009-03-18 -- wget手册 wget命令使用方法 (1)
2008-07-25 -- Californication (2)
2007-11-23 -- dict.TCL.install (0)
2008-10-14 -- Cloud Computing展望 (5)
2005-12-24 -- 搞笑的平安夜 (6)
2008-12-02 -- Cloud Computing Report (0)

历数云计算为安全带来的七大益处

jimey — Sun, 18 Jan 2009 06:13:59 +0000

二、事件快速反应
取证准备：在必要的时候，我可以利用基础架构即服务(Infrastructure-as-a-Service，简称IaaS)供应商提供的条件，为自己公司建立一个专门的取证服务器。当事件发生需要取证时，我只需要支付在线存储所产生的费用。而不需要额外配置人员去管理远程登陆及其软件，而我所要做的，就是点击云提供商Web界面中的一些按钮即可。如果一旦产生多个事件反应，我可以先复制一份，并把这些取证工作分发到不同部门或者人员手中，然后进行快速分析并得出结论。不过，为了充分发挥这项功能，取证软件供应商需要由过去传统的软件授权许可转变到新型网络许可模式。
缩短取证时间：如果有某个服务器在云中出现了故障，我只要在云客户端点击鼠标，克隆该服务器并使得克隆后的服务器磁盘对取证服务器开放。我根本不需要临时寻找存储设备，并花时间等待其启动并进入使用状态，从而，大大缩短取证时间。
降低服务器出错概率：和刚才讲述的情况类似，即使有某台服务器出现故障，也可以在极短时间内，快速克隆并拥有全新的服务器供使用。另外，在某些情况下，更换出故障的硬件也不会影响到取证的正常进行。
取证更有针对性：在同一个云中，拥有克隆服务器的速度会快很多——克隆服务器可以更快的速度分发云提供商专门设计的文件系统。如果从网络流量角度来看的话，在同一个云中的服务器副本，可能并不会产生额外的费用。而如果没有云的话，要实现同样的目的，需要花费大量宝贵的时间和昂贵的硬件成本。在云环境下，我只需要对有用的取证支付存储费用。
隐藏取证痕迹：有一些云存储可以执行加密校验和散列( hash)。比如，Amazon S3 会在你存储数据的时候自动生成一个MD5 散列(hash)。在理论上，你也并不需要浪费时间去使用外部工具生成MD5加密校验，因为云已经完全具备这些功能。
缩短存取受保护数据时间：现在CPU性能已经十分强大。保护数据的密码，需要花费很长时间来检验，而现在云环境下配置强大的CPU，可以在短时间更大范围内检验出保护数据的密码性能。从而，批量处理受保护数据的存取工作也会变得简易快速.

[via]

红色部分，之前很久之前看的一篇文章一直在反复说hash，之前也不是很想明白究竟有什么意义，暂时这边看到一个文章说到了hash的作用之一，hm 暂时丢出来记录下

无关博文

2008-07-09 -- 实验室继续调整 (0)
2009-02-07 -- Sigur Ros – We Play Endlessly (0)
2008-08-21 -- 【转帖】全国上下齐讽A队，一片和谐 (0)
2008-10-31 -- Clonezilla – Clone System Ubuntu (0)
2005-10-13 -- 从现在开始使用这个bloger。。。看起来蛮不错的 (0)
2008-12-14 -- A deaf girl learns to play violin – An Inspiring must view (0)
2009-02-14 -- Cloud Computing defined by Berkeley RAD Labs (0)
2008-11-17 -- Indie Pop & Post Rock (1)
2008-11-11 -- 배빼로날 축하해. 1111纪念下 (0)
2008-08-24 -- QualNet教程中文版 (9)
2010-07-03 -- 首尔第二天 (1)
2008-09-12 -- Dropbox (0)
2009-02-09 -- 如何解决MSN出现8100030d错误 (0)
2009-11-08 -- NATE ON 某新功能?or 我以前没发现的功能 (1)
2009-03-27 -- 同步twitter到饭否 (0)

Jimey’s Life && Jimey’s World » Cloud Computing

Ubuntu10.04 编译hadoop-${version}-eclipse- plugin.jar and hadoop-0.20.3-dev-eclipse-plugin.jar

相关博文

几篇关于twitter,hadoop,Cassandra,Pig,FlockDB的slide

相关博文

云端医疗信息系统的不成熟构思

相关博文

Karmasphere Studio for Hadoop

相关博文

Eclipse+Hadoop 全linux环境开发

相关博文

编译hadoop-${version}-eclipse-plugin.jar and hadoop-0.20.1-eclipse-plugin.jar download

相关博文

hadoop-0.20.0

相关博文

Hadoop+MySQL结合

相关博文

[ZT]Writing your own real-time search engine and Adwords service

相关博文

几个分享和想法

相关博文

windows环境下eclipse调试远端ubuntu环境的hadoop

相关博文

Linux server clustered HDFS: access from Windows eclipse Java application

Re: Linux server clustered HDFS: access from Windows eclipse Java application

Confusing NameNodeFailover page in Hadoop Wiki

Re: Linux server clustered HDFS: access from Windows eclipse Java application

Re: Confusing NameNodeFailover page in Hadoop Wiki

Re: Confusing NameNodeFailover page in Hadoop Wiki

相关博文

基于Eclipse的Hadoop应用开发环境的配置

相关博文

Big Data: Technologies and Techniques for Large-Scale Data

相关博文

Zotero 1.5 Beta Released: Join Us In The Clouds

Zotero 1.5 Beta Released: Join Us In The Clouds

相关博文

Cloud Computing defined by Berkeley RAD Labs

Executive summary:

相关博文

How Businesses Can Use P2P

What’s in a Name?

P2P and the Cloud

P2P as a Social Tool

So, Show Us the Money

相关博文

Cisco in The Cloud: Application Extension Platform (AXP)

The Networking Problems In Branch Offices

相关博文

软件算法可视化大型数据集

无关博文

历数云计算为安全带来的七大益处

无关博文