Cannot obtain block length for LocatedBlock故障分析和解决

摘要: 作者：Syn良子 出处：http://www.cnblogs.com/cssdongl 转载请注明出处

这几天想cat一下某天的HDFS文件内容的时候突然报Cannot obtain block length for LocatedBlock异常，get也一样，这样无法访问hdfs文件的问题必须解决，Mark一下问题背景和解决过程

一.问题背景

问题产生的原因可能是由于前几日Hadoop集群维护的时候，基础运维组操作不当，先关闭的Hadoop集群，然后才关闭的Flume agent导致的hdfs文件写入后状态不一致。排查和解决过程如下.

二.解决过程

1.既然是hdfs文件出问题,用fsck检查一下吧

hdfs fsck /

当然你可以具体到指定的hdfs路径,检查完打印结果没有发现任何异常，没有发现损坏或者Corrupt的block，继续排查

2.那么加上其他参数细查

hdfs fsck / –openforwrite

ok,这次检查出来不少文件打印显示都是 openforwrite状态,而且我测试相应文件确实不能读取,这很不正常不是吗？Flume已经写过的hdfs文件居然还处于openforwrite状态，而且无法cat和get

所以这里的”Cannot obtain block length for LocatedBlock”结合字面意思讲应该是当前有文件处于写入状态尚未关闭，无法与对应的datanode通信来成功标识其block长度.

那么分析其产生的可能性，举栗子如下

1>Flume客户端写入hdfs文件时的网络连接被不正常的关闭了

或者

2>Flume客户端写入hdfs失败了，而且其replication副本也丢失了

我这里应该属于第一种，总结一下就是Flume写入的hdfs文件由于什么原因没有被正常close，状态不一致随后无法正常访问.继续排查

3.推断:HDFS文件租约未释放

可以参考这篇文章来了解HDFS租约机制 http://www.cnblogs.com/cssdongl/p/6699919.html

了解过HDFS租约后我们知道,客户端在每次读写HDFS文件的时候获取租约对文件进行读写，文件读取完毕了，然后再释放此租约.文件状态就是关闭的了。

但是结合当前场景由于先关闭的hadoop集群，后关闭的Flume sink hdfs,那么hadoop集群都关了，Flume还在对hdfs文件写入，那么租约最后释放了吗？答案是肯定没释放.

4.恢复租约

对于这些状态损坏的文件来讲，rm掉的话是很暴力的做法，万一上游对应日期的数据已经没有rention呢？所以，既然没有释放租约，那么恢复租约close掉文件就是了,如下命令

hdfs debug recoverLease -path <path-of-the-file> -retries <retry times>

请将<path-of-the-file>修改成你需要恢复的租约状态不一致的hdfs文件的具体路径,如果要恢复的很多，可以写个自动化脚本来找出需要恢复的所有文件然后统一恢复租约.

ok，执行完命令后再次cat对应hdfs文件已无异常，顺利显示内容，问题解决.

Cannot obtain block length for LocatedBlock故障分析和解决

标签：文件   需要   lease   debug   暴力   恢复   标识   pat   脚本   

小编还为您整理了以下内容，可能对您也有帮助：

唯美英语短文20篇（加分）

1、

Reading is a happy thing. For those who love reading, once they read it,

they can't stop, they can't give up reading, they want to read to the

ends of the world. Some people say: "the deepest and most peaceful

happiness Calm down, open the book, the deposition

of ink a little bit over the space, those sty happiness a little bit

open. Reading is really fun.

读书是一件快乐的事情。对于爱读书的人一但读上就让人欲罢不能，无法放弃读书，想让人读到天涯海角。有人说：“人生最深最平和的快乐，就是静观天地与人生，慢慢品味出它的和谐与美。”静下心来，翻开书本，那些沉积的墨香一点点溢满空间，那些尘封的快乐一点点打开。读书真的很快乐。

Goethe once said, "to read a good book is to talk to a noble man." Yes, reading good books is like making good teachers and friends, which benefits me a lot; Reading good books is like bathing in the warm spring breeze, Reading good books is like finding the source of happiness, which makes me understand the true meaning of happiness.

歌德以前说过：“读一本好书，就是和一位高尚的人谈话。”是的，读好书，就像结交了良师益友，使我受益匪浅；读好书，就像沐浴在温暖的春风里，使我心旷神怡；读好书，就像找到了快乐之源，使我领悟到快乐的真谛。

2、

Life is picturesque, there are thick ink splashing, there are light pen light description; Life is like a song, there are light chanting and shallow singing, there are also Huang zhongdalu. You don't see the life of Li Tai and Du Gongbu. They have created the greatness of life; Pei Duofei, Shelley's life, light pen light description, finally write the true meaning of life.

生命如画，有浓墨泼洒，也有淡笔轻描；生命如歌，有轻吟浅唱，也有黄钟大吕。君不见李太白、杜工部之一生，浓墨泼洒，铸就生命之伟大；裴多菲、雪莱之一生，淡笔轻描终写尽生命之真谛。

You can't see Tao Yuanming's "picking chrysanthemums under the East fence, leisurely seeing Nanshan" and Tan Sitong's "I smile from the horizontal knife to the sky, leaving my liver and gall in Kunlun".

君不见轻音浅唱之陶渊明“采菊东篱下，悠然见南山”，高歌猛进之谭嗣同“我自横刀向天笑，去留肝胆两昆仑”。

3、

Spirit is a monument, a cornerstone, a beacon on the way forward, and a yardstick for a brilliant life. With modesty, rich and talented, Confucius slowly chanted that "if three people walk, there must be a teacher." he was not ashamed to ask questions, and eventually became a generation of Confucian masters.

精神是一座丰碑，是一块基石，是前进道路上的灯塔，是灿烂人生的标尺。有了谦虚，学富五车、才高八斗的孔子缓缓吟出“三人行，必有我师”，不耻下问，终成一代儒学大师。

With tolerance, ambition and high spirited, Duke Huan of Qi abandoned his personal feud, appointed people on the basis of merit, achieved great achievements for thousands of years, and finally never threw away the Han envoys' festival in his hand. He cared for his motherland and eventually became a patriotic model for thousands of years.

有了宽容，雄心勃勃、意气风发的齐桓公摈弃私仇，任人唯贤，成就千秋伟业，终成一代霸主；有了坚贞，寂寞忧愁、孤寂郁愤的苏武始终不扔掉手中的汉使族节，心系祖国，终成千秋爱国楷模；有了奉献，善良质朴、平凡无闻的徐虎才能够十年如一日，情洒百姓，温暖万家，终成当今的“活雷锋”。

Spirit, through the tunnel of history, across the long river of history, will forever support the towering and spectacular building of life.

精神，穿越历史隧道，趟过历史长河，将永久撑起人生巍峨壮观的大厦。

4、

Night, like ink drops on the canvas of the sky, permeates and faints. With a few strokes of the brush, there are stars. With a few strokes, the soft moonlight will shoot into your heart.

夜，如墨般滴在天空的画布上，渗透、晕开，用画笔轻点几下，便有了星辰，再描上几下，柔和的月光将射进你的心房。

The firefly's light is as warm as the stars. The cricket took out his violin and played a beautiful waltz with the breeze.

萤火虫的光足以媲美点点繁星，却要温暖几分。蟋蟀拿出小提琴，与微风合作，演奏出一曲优美的圆舞曲。

The stream is flowing slowly, like the conveyor belt of time, passing by a little bit. Flowers even at night will also send out fragrance, the fish in the pool spit bubbles, bubbles are curtain after curtain of dreams. The sky began to rain, one by one, bright to the extreme, beautiful and sad, as if that is the sky tears.

溪水缓缓地流着，像时间的输送带，一点点流逝。花朵就算在夜晚也会散发芳香，池里的鱼儿吐着泡泡，泡泡里是一帘又一帘的美梦。天边下起了雨，一颗又一颗，明亮至极，秀丽且悲伤，仿佛那就是天空的眼泪。

Gently pull the curtain, is afraid to disturb that quiet; Lie down gently for fear of waking the sleeping stars; Gently cover the quilt, is afraid to break the sweet country autumn dream.

轻轻拉上窗帘，是怕惊扰那份宁静；轻轻地躺下，是怕吵醒那些熟睡的星星；轻轻地盖上被子，是怕打破那甜美的乡下秋梦。

5、

Human life is a chain. There will always be children younger than you and old people older than you. Each of us has his own position. It is a financial report that no one can take away.

人的生命是一根链条，永久有比你年轻的孩子和比你年迈的老人。我们每个人都有自己的位置，它是一宗谁也掠不去的财报。

Don't worry about when you are young or old. As long as we live a day, the wealth of youth glitters. There is only one dark night that can cover its light, that is, you think you are old.

不要计较，何时年轻，何时年老。只要我们生存一天，青春的财富就闪闪发光。能够遮蔽它的光芒的暗夜只有一种，那就是你自以为已经衰老。