Al Shabaab is surviving because it is paying more attention to money than mayhem.
Al Shabaab leaders have always paid attention for finances, something unsuccessful Islamic terror groups generally put less effort into.
It's a Somali specialty to pay attention to the financial opportunities and that is one of the factors that fuels corruption in the government.
All that foreign aid is considered fair game for whoever can grab it. Donor efforts to reduce the corruption often meets with indifference masked by insincere agreement to cooperate. The money keeps disappearing before getting to where the donor specified it should go. The looting of the donor funds has become so lucrative that even al Shabaab is recruiting government financial officials to aid the Islamic terrorist efforts to get a share of the looted aid funds.
Many Somalis realize that al Shabaab is unlikely to gain control of the government or most of the country. But as a determined and heavily armed criminal gang the financial opportunities are enormous. It's like the ancient Chinese description of a government bureaucracy; "the object is not to win or lose but to keep the game going." Among terrorists who have survived for a while and operated in different countries, al Shabaab has a reputation of being more mercenary than religious zealot. This is one reason al Shabaab has been hostile tp foreign Islamic terrorists, who are more likely to find acceptance in the smaller, and much less prosperous and successful Somali ISIL (Islamic State in Iraq and the Levant) faction.
The Somali Army proved capable of pushing al Shabaab out of towns beyond Mogadishu and Kismayo (the major city and port further south) but the government has not been able to establish local government services like security and justice administration.
This meant working out equitable arrangements with local clan leaders. These local elders were often crooks themselves but they were local crooks who had reached accommodations with the local population. Al Shabaab offered to cooperate because they tried it the other way and it did not work out. Taking Islamic radicalism to the extreme tramples on too many local customs and norms and never works in the long run. The federal government could have worked out deals with the locals to replace al Shabaab control with federal control but that did not happen. Army leaders weren't equipped or authorized to do this sort of thing and federal official were more interested in looting opportunities than in restoring administrative and government services the locals would accept. As a result the army and peacekeeper efforts to maintain control of these areas is gone as soon as the large concentration of troops moves on.
Patience As Run Out
Meanwhile the clock is ticking on how long the government will have foreign military assistance. Many foreign aid efforts in Somalia have been shut down because of the persistent, and extreme, corruption. That's the main reason the 21,000 man peacekeeping force is going to start withdrawing in 2021.
Fresh fears of fighting in Galmugdug after fall out between regional state and Ahlu Sunnah Wal Jama'ah, in the central...
It is mainly about money and limited resources. Peacekeepers are expensive and over a decade of peacekeeping in Somalia has produced meager results. There is demand for peacekeepers in other parts of the world, often in places where peacekeepers make more of a difference.
Somalia is considered a wasted effort in a world of too much demand and too little supply. This is a common situation with failed states and Somalia is the worst of the worst. The Americans and UN have concluded that a more effective way to deal with Somalia is to pull out the expensive peacekeepers and provide more assistance to Somali neighbors Ethiopia, Kenya and Djibouti to deal with al Shabaab and Somali misbehavior in general.
The UN and AU (African Union) have been supplying Somali with peacekeepers and the money to pay and sustain them since 2007. Back then the plan was for 8,000 peacekeepers who would only be needed for six months. That force did not disappear by the end of 2007 but kept growing and quickly reached 22,000. It made some difference, but in the face of massive corruption in the Somali government and various Somali communities that demanded help from the peacekeepers, the operation proved far more expensive and time-consuming than expected.
The peacekeepers are all Africans from neighboring countries and their commanders understand the violent nature of Somalia, which has been sending raiders into neighboring areas for centuries. Somalis are more accepting of this sort of thing than the neighbors. That may explain why Somalia can be so corrupt while also have a population that considers themselves relatively well off.
The American Response
U.S. SOCOM (Special Operations Command) and CIA intelligence gathering operations have local facilities in Djibouti (where the aircraft and some SOCOM personnel are stationed).
There is a smaller satellite facility in Mogadishu, where a heavily guarded compound houses CIA and SOCOM operators who organize and run local intelligence operations and distribute information to peacekeepers, Somali military and trusted local allies.
Details of how the intel collection operation works are kept out of the news but it does explain why al Shabaab targets Somali and foreign intel officials whenever possible. The intel and airstrike operations have been very effective and are one reason why there are now so few al Shabaab attacks in general and a growing list of al Shabaab defeats and retreats. But al Shabaab has adapted. This is a pattern that is by design not chance. Al Shabaab does not mention this sort of thing in their online propaganda but among themselves it is a major topic of conversation and frustration. The U.S. recognizes the difficulty of operating in Somalia and refuses to station many military or intel personnel there. There are currently about 500 American troops in Somalia and only 5,000 (down from 7,500 in 2018) in all of Africa. These belong to AFRICOM and most of those troops are based in Djibouti, Somalia's northern neighbor. The United States considers Russia, China and Iran as more important than local problems in corrupt corners of Africa. Somalia is considered low priority and the U.S. has concluded that the locals, like neighboring Djibouti, Ethiopia and Kenya are capable to dealing with Somali aggression and invasion