Linux server clustered HDFS: access from Windows eclipse Java application

貌似有人也有一样的问题，找不到hadoop.job.ugi

不过我也找不到。。搞了3个小时实在是奇怪

还是用svn put上去好了。。

正常的配置是

You will now be asked to fill in a number of parameters identifying the server. To connect to the VMware image, the values are:

Location name: (Any descriptive name you want; e.g., "VMware server")Map/Reduce Master Host: (The IP address printed at startup)Map/Reduce Master Port: 9001DFS Master Port: 9000User name: hadoop-user

Next, click on the "Advanced" tab. There are two settings here which must be changed.

Scroll down to hadoop.job.ugi. It contains your current Windows login credentials. Highlight the first comma-separated value in this list (your username) and replace it with hadoop-user.

Next, scroll further down to mapred.system.dir. Erase the current value and set it to /hadoop/mapred/system.

When you are done, click "Finish." Your server will now appear in the Map/Reduce Locations panel. If you look in the Project Explorer (upper-left corner of Eclipse), you will see that the MapReduce plugin has added the ability to browse HDFS. Click the [+] buttons to expand the directory tree to see any files already there. If you inserted files into HDFS yourself, they will be visible in this tree.

[via]

但是目前比较奇怪找不到这个hadoop.job.ugi无法显示。。

by Alberto Forcén Aug 05, 2008; 06:16pm :: Rate this Message: – Use ratings to moderate (?)

Reply | Reply to Author | Print | View Threaded | Show Only this Message

Hi all.

I’m running a clustering HDFS on linux and I need to access files (I/O) from eclipse Java application running on Windows. It seems simple, but is it possible?

I have write code using API but I have a problem: when code invokes DistributedFileSystem.initialize() method I receive an exception: java.net.SocketTimeoutException

[code]

String ipStr = "192.168.75.191";

String portStr = "9000";

String uriStr = "http://" + ipStr + ":" + portStr;

Configuration conf = new Configuration();

conf.set("hadoop.job.ugi", "user,group"); // Usuario y grupos a los que pertenece

DistributedFileSystem dfs = new DistributedFileSystem();

dfs.initialize(new URI(uriStr), conf);

[/code]

[trace]

Exception in thread "main" java.net.SocketTimeoutException: timed out waiting for rpc response

at org.apache.hadoop.ipc.Client.call(Client.java:559)

at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)

at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)

at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)

at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)

at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)

at org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)

at examples.HadoopDFS.main(HadoopDFS.java:153)

[/trace]

______________________________________________

Enviado desde Correo Yahoo! La bandeja de entrada más inteligente.

Re: Linux server clustered HDFS: access from Windows eclipse Java application

by Qin Gao-2 Aug 05, 2008; 10:13pm :: Rate this Message: – Use ratings to moderate (?)

Reply | Reply to Author | Print | View Threaded | Show Only this Message

I think IBM has a plugin that can access HDFS, I don’t know whether it

contains source code, but maybe it helps.

www.alphaworks.*ibm*.com/tech/mapreducetools

On Tue, Aug 5, 2008 at 5:16 AM, Alberto Forcén <otrebla81@…> wrote:

> Hi all.

>

> I’m running a clustering HDFS on linux and I need to access files (I/O)

> from eclipse Java application running on Windows. It seems simple, but is it

> possible?

> I have write code using API but I have a problem: when code invokes

> DistributedFileSystem.initialize() method I receive an exception:

> java.net.SocketTimeoutException

> [code]

> String ipStr = "192.168.75.191";

> String portStr = "9000";

> String uriStr = "http://" + ipStr + ":" + portStr;

> Configuration conf = new Configuration();

> conf.set("hadoop.job.ugi", "user,group"); // Usuario y grupos a los que

> pertenece

> DistributedFileSystem dfs = new DistributedFileSystem();

> dfs.initialize(new URI(uriStr), conf);

> [/code]

> [trace]

> Exception in thread "main" java.net.SocketTimeoutException: timed out

> waiting for rpc response

> at org.apache.hadoop.ipc.Client.call(Client.java:559)

> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)

> at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)

> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)

> at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)

> at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)

> at

> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)

> at examples.HadoopDFS.main(HadoopDFS.java:153)

> [/trace]

> ______________________________________________

> Enviado desde Correo Yahoo! La bandeja de entrada más inteligente.

… [show rest of quote]

Confusing NameNodeFailover page in Hadoop Wiki

by Konstantin Shvachko Aug 06, 2008; 03:42am :: Rate this Message: – Use ratings to moderate (?)

Reply | Reply to Author | Print | View Threaded | Show Only this Message

I was wondering around Hadoop wiki and found this page dedicated to name-node failover.

http://wiki.apache.org/hadoop/NameNodeFailover

I think it is confusing, contradicts other documentation on the subject and contains incorrect facts. See

http://hadoop.apache.org/core/docs/current/hdfs_user_guide.html#Secondary+Namenode

http://wiki.apache.org/hadoop/FAQ#7

Besides it contains some kind of discussion.

It is not that I am against discussions, lets have them on this list.

But I was trying to understand were all the confusion about secondary-node issues comes from lately…

Imho we either need to correct it or remove.

Thanks,

–Konstantin

Re: Linux server clustered HDFS: access from Windows eclipse Java application

by Michael Bieniosek Aug 06, 2008; 03:56am :: Rate this Message: – Use ratings to moderate (?)

Reply | Reply to Author | Print | View Threaded | Show Only this Message

Are you sure both ends are running the same version of hadoop?

On 8/5/08 2:16 AM, "Alberto Forcén" <otrebla81@…> wrote:

Hi all.

I’m running a clustering HDFS on linux and I need to access files (I/O) from eclipse Java application running on Windows. It seems simple, but is it possible?

I have write code using API but I have a problem: when code invokes DistributedFileSystem.initialize() method I receive an exception: java.net.SocketTimeoutException

[code]

String ipStr = "192.168.75.191";

String portStr = "9000";

String uriStr = "http://" + ipStr + ":" + portStr;

Configuration conf = new Configuration();

conf.set("hadoop.job.ugi", "user,group"); // Usuario y grupos a los que pertenece

DistributedFileSystem dfs = new DistributedFileSystem();

dfs.initialize(new URI(uriStr), conf);

[/code]

[trace]

Exception in thread "main" java.net.SocketTimeoutException: timed out waiting for rpc response

at org.apache.hadoop.ipc.Client.call(Client.java:559)