azure - C# Map Reduce failing with "{"Response status code does not indicate success: 403 (Forbidden)."} sometimes 401: credentials required -
an unhandled exception of type system.aggregateexception
occurred in mscorlib.dll
inner exception: {"response status code not indicate success: 403 (forbidden)."}
sometime get: {"response status code not indicate success: 401 (credentials required)."}
all logins correct. hadoop.connect() connect properly.
stack trace:
at system.threading.tasks.task.throwifexceptional(boolean includetaskcanceledexceptions) @ system.threading.tasks.task.wait(int32 millisecondstimeout, cancellationtoken cancellationtoken) @ system.threading.tasks.task.wait() @ microsoft.hadoop.webclient.webhcatclient.webhcatmapreducestreamingexecutor.execute(boolean throwonerror) @ microsoft.hadoop.mapreduce.execution.hadoop.streamingjobexecutorbase.executecore(type mapper, type reducer, type combiner, hadoopjobconfiguration config)
@ microsoft.hadoop.mapreduce.execution.hadoop.streamingjobexecutorbase.execute(type mappertype, type reducertype, type combinertype, hadoopjobconfiguration config) @ microsoft.hadoop.mapreduce.execution.hadoop.streamingjobexecutorbase.execute[tmapper,treducer](hadoopjobconfiguration config) @ pwc.demo.hdpclient.program.main(string[] args) in c:\asc project info\talentexchange\demo project\pwc.demo.hdpclient\pwc.demo.hdpclient\program.cs:line 49 @ system.appdomain._nexecuteassembly(runtimeassembly assembly, string[] args) @ system.appdomain.executeassembly(string assemblyfile, evidence assemblysecurity, string[] args) @ microsoft.visualstudio.hostingprocess.hostproc.runusersassembly()
@ system.threading.threadhelper.threadstart_context(object state)
@ system.threading.executioncontext.runinternal(executioncontext executioncontext, contextcallback callback, object state, boolean preservesyncctx) @ system.threading.executioncontext.run(executioncontext executioncontext, contextcallback callback, object state, boolean preservesyncctx) @ system.threading.executioncontext.run(executioncontext executioncontext, contextcallback callback, object state) @ system.threading.threadhelper.threadstart()
class program { static void main(string[] args) { hadoopjobconfiguration myconfig = new hadoopjobconfiguration(); myconfig.inputpath = "asv://hdsto-something-.net/data/in/reviews.txt"; myconfig.outputfolder = "asv://hd-something-.blob.core.windows.net/data/out/"; environment.setenvironmentvariable("hadoop_home", @"c:\apps\dist\hadoop-2.7.1.2.3.3.1-25\bin"); environment.setenvironmentvariable("java_home", @"c:\program files\java\jre1.8.0_101\bin\javaw.exe"); uri azurecluster = new uri("https://somename--demo.azurehdinsight.net"); string clusterusername = "***"; string clusterpassword = "****"; // name of account under hadoop execute jobs. // "hadoop". string hadoopusername = "hadoop"; // azure storage information. string azurestorageaccount = "somestore.blob.core.windows.net"; string azurestoragekey = "***=="; string azurestoragecontainer = "**"; bool createcontinerifnotexist = false; ihadoop mycluster = hadoop.connect(azurecluster, clusterusername, hadoopusername, clusterpassword, azurestorageaccount, azurestoragekey, azurestoragecontainer, createcontinerifnotexist); //execute mapreduce job mapreduceresult jobresult = mycluster.mapreducejob.execute<mysimplemapper, mysimplereducer>(myconfig); int exitcode = jobresult.info.exitcode; string exitstatus = "failure"; if (exitcode == 0) exitstatus = "success"; exitstatus = exitcode + " (" + exitstatus + ")"; console.writeline(); console.write("exit code = " + exitstatus); console.read(); } } public class mysimplemapper : mapperbase { public override void map(string inputline, mappercontext context) { int value = int.parse(inputline); string key = (value % 2 == 0) ? "even" : "odd"; context.emitkeyvalue(key, value.tostring()); } } public class mysimplereducer : reducercombinerbase { public override void reduce(string key, ienumerable<string> values, reducercombinercontext context) { //initialize counters int mycount = 0; int mysum = 0; //count , sum incoming values foreach (string value in values) { mysum += int.parse(value); mycount++; } context.emitkeyvalue(key, mycount + "t" + mysum); } }
your input path should not specific file, directory. directory should contain files , not folders.
myconfig.inputpath = "asv://hdsto-something-.net/data/in/";
Comments
Post a Comment