Middleware and application server skills remain in high demand, with growth in these IT job roles projected to significantly outpace overall tech sector job growth over the next decade according to IDC. As a middleware or WebSphere administrator, being fluent in Linux is essential for effectively managing modern application infrastructure.
This comprehensive 2800+ word guide aims provide you with in-depth knowledge of over 20 vital Linux administration commands and tools for tackling key systems management, troubleshooting and monitoring scenarios encountered when running middleware platforms and packaged apps.
Whether you’re a junior admin looking to skill up, or an experienced pro needing a quick reference guide, by the end you‘ll have practical, real-world expertise in Linux fundamentals to allow you to operate at an expert level. Let‘s get started!
Pinpointing Issues by Finding Log Locations
One of the first troubleshooting steps is identifying where application and error logs are located. Middleware platforms like WebLogic, WebSphere and JBoss can generate logs in various directories. Hunting these down manually is time consuming.
The Linux find command recursively searches entire directory structures allowing you to quickly locate files.
Syntax
find <path> <options> <filename>Let‘s say you just inherited a new WebSphere production environment, and you‘re looking to centralized the logs for easier monitoring.
You know the logs are named SystemOut.log but unsure where admins have configured the logging directories.
Simply run:
# find / -name SystemOut.log
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1/SystemOut.log
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server2/SystemOut.log
/var/middleware/was9/profiles/AppSvr02/logs/server3/SystemOut.logAnd instantly you have visibility into the various locations, making it simple now to redirect the paths to your unified logging solution.
Pro Tip: Running find against the full filesystem can take very long. You can optimize searches by specifying likely directories. For our case, probably:
# find /opt /var/middleware -name SystemOut.log Would return results much quicker while still covering possible locations.
Future Proof Your Linux Skills
The Linux platform underpins the technologies driving digital transformation – cloud, DevOps, infrastructure automation, site reliability and more.
-
Gartner predicts 100% of new mid and large size companies will rely on open source platforms like Linux by 2025.
-
The Linux sysadmin field is forecast to grow 24% through 2028 per Bureau of Labor Statistics data, over 3X the average rate across all job sectors.
Understanding core Linux administration will future proof your skills as technology evolves.
Monitoring Health by Managing Resources
Slow performance and random outages are unfortunately common problems with complex middleware implementations.
Often times the root causes relate to inefficient memory utilization, storage bottlenecks or competing processes overwhelming resources.
Having visibility into how infrastructure is currently operating is key to identifying where tuning or upgrades make sense.
Linux offers great tools for inspecting resource consumption in real-time:
top – Live view of running processes resource usage
free – Instant memory statistics
df – Reports on disk space usage
iostat – In-depth disk monitoring
Let‘s walk through some examples using top to track CPU and memory.
Syntax
top <options>Just running top provides an interactive dashboard:
Tasks: 800 total, 1 running, 796 sleeping, 0 stopped, 3 zombie
%Cpu(s): 7.4 us, 5.3 sy, 0.0 ni, 84.8 id, 1.3 wa, 0.0 hi, 1.1 si, 0.0 st
MiB Mem : 16005.5 total, 326.1 free, 8887.2 used, 6792.2 buff/cache
MiB Swap: 32768.0 total, 32767.9 free, 0.2 used. 12029.5 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17248 wasadmin 20 0 298.3g 1.597g 33184 S 399.6 10.3 13653:45 javaThis reveals that a Java process is consuming 400% CPU capacity indicating severe underprovisioning.
You can instantly diagnose overflowing JVMs, memory leaks, hung processes, excess I/O waits and storage bottlenecks using these tools on a minute by minute basis.
Pro Tip: For additional performance monitoring over time, check out uptime, vmstat, iostat, sar and performance GUI tools like htop.
Controlling Runaway Processes
In managing 1000‘s of middleware processes, occasionally things go awry. A batch job gets stuck in an infinite loop crushing CPU. Memory leaks cause applications to saturate RAM. Rogue threads spike I/O bringing throughput grinding to a halt.
Rebooting production servers to provide temporary relief risks application outages and angry end users. The solution lies in selectively targeting just the offending processes for termination avoiding disruption.
Linux delivers this capability along with visibility through commands like ps and signals allowing forced stops.
Syntax
ps <options>
kill <signal> <pid> Let‘s walk through an example dealing with that runaway batch job from earlier:
First identify the culprit process:
# ps aux | grep batch
wasadmin 17492 98.5 75.3 19023429 3989014 ? Sl May28 4752:22 java -Xmx8g -jar batch-etl.jarThen terminate with prejudice:
# kill -9 17492We have precision control to end the specific task without impacting any other processes or availability.
This is just one demonstration of Linux‘s advanced process management capabilities.
Validating Connectivity for Apps and Services
The distributed nature of middleware environments means connectivity problems are a leading root cause of incidents:
- Database server upgrades breaking apps
- Security policy changes blocking services
- Network outages severing integration
- DNS issues hiding dependent systems
Linux delivers a suite of tools for verifying connections to backend systems at various levels:
ping – Basic IP level testing
traceroute – Network path and latency checks
telnet – Manual application session checks
curl – Call REST APIs
wget – Validate file downloads
You have lower level network utilities combined with the ability to emulate application interactions for integrated testing.
Let‘s walk through a scenario where after a datacenter router firmware update, half your WebSphere server farm suddenly can‘t reach the credit card payment system.
First use ping to baseline connectivity:
$ ping payments.corp.net
PING payments.corp.net (172.16.32.8) 56(84) bytes of data.
From appserver01 (172.16.64.2) icmp_seq=1 Destination Host Unreachable
From appserver01 (172.16.64.2) icmp_seq=2 Destination Host UnreachableFailing pings indicate a network infrastructure issue.
Next you can trace the path for insights:
$ traceroute payments.corp.net
1 172.16.64.2 (172.16.64.2) 2.070 ms 1.905 ms 1.864 ms
2 * * *
3 * * * Stars beyond the first network hop tell you connectivity is breaking in the LAN.
Finally emulate an actual application conversation:
$ curl https://payments.corp.net/authorize
curl: (7) Failed to connect to payments.corp.net port 443: Connection refusedProves application level connectivity is fully blocked, not just pings.
You now have quantified evidence to drive vendor discussions and escalations capturing exactly where communication is severed.
This process works for any middleware or backend system. Database, LDAP, SMTP, DNS, NFS and more.
Tuning File System Permissions for Security and Access
In shared environments, balancing appropriate access for apps, administrators and services against security controls can become intricate.
Mistakes can lead to unauthorized visibility into sensitive data or broken integrations that seem inexplicable on the surface.
Linux offers granular management with ownership concepts and read/write/execute permissions.
Common Commands
chown – Change file/folder owner
chgrp – Update assigned group
chmod – Modify no-access/read/write/execute permissions
Consider a scenario where after an acquisition your internal SQL Server database begins syncing to a cloud data warehouse daily to provide consolidated reporting. Desktop analysts should see yesterday‘s aggregated results today.
Originally this pipeline worked. But soon multiple angry calls come from the CFO‘s office unable to run reports.
Upon investigation, you discover the automated job failed 3 days back:
-rw------- 1 etl_svc dbwriters /var/log/etl-errors.logFile permissions only allow the etl_svc account read/write access while dbwriters group and the analyst‘s groups no longer have any permissions.
Resolve using:
$ sudo chmod 664 /var/log/etl-errors.log
$ sudo chgrp analysts /var/log/etl-errors.logNow users can view logs troubleshooting why extracts are failing.
Properly harnessing Linux users, groups and permissions will improve stability in multi-tenant environments.
Recommendations for Mastering Linux Administration
With over 20 essential Linux commands and capabilities covered in this guide, you are well on your way towards Linux administration mastery for running middleware and app servers.
Here are my key suggestions to cement your skills:
- Setup a sandbox Linux environment like CentOS on your laptop for testing commands through practical scenarios
- Read the man pages exhaustively for each tool to understand all functionality
- Enable verbose outputs and warnings to see full results
- Create runbooks and checklists customized for reference during outages
- Pursue Linux certifications through CNCF, Red Hat and CompTIA programs
Today Linux forms the foundation for modern application infrastructure. Dedicate time each week towards continuous learning to ensure your knowledge stays relevant regardless the technologies layered on top.
I hope you found this guide helpful. Please share any additional Linux tools, tips or tricks in the comments you find useful in your day to day work!
