How to get real time log analysis
My application constructs a lot of logs every day.  I used to use crontab and some Java application to store these data into MySQL. Otherwise I have a manage system , and its functions mainly based on the analysis of these data in MySQL. My application is not running on a single web-app server. So it is not a good way to analyze data based on the log files directly. But as the log file is bigger and bigger, it's a long time for crontab to store log files completely, I cannot have a real time analysis. So what optimization can be done to get a real time analysis?

If your application is a java application and you use log4j you may use a jdbc Appender to log direct to the database.

You can use OtrosLogViewer in "Batch processing" mode. I have added to cron scaning job of my logs every day at midnight. I'm also using Log4j DailyRollingFileAppender
Disclaimer: I am the author of OtrosLogViewer

i think you can create bunch of log files for each run rather than creating single file and writing all the errors,exception into it.
i face the same problem as for my java based application the size of log file it much big.
so i created a folder and stored files in below mention form
< FolderName + DateOfRun >
              -----< File Contains Exception/Error for core Logic >.txt
              -----< File Contains Exception/Error for primary Logic >.txt
              -----< File Contains Exception/Error for secondary Logic >.txt
              -----< File Contains Exception/Error for miscellaneous  Logic >.txt

this provides me better way to handle huge data after when my appli. completes its run by taking almost 4hrs-5hrs.

We use XpoLog as our log analysis system it is a platform that allow real time analysis of log files without agents.  
