How to Read a Log File of 1 Tb in Jav

Reading LogRecords From A Log File or Table

The log writing sample programme LogSample.java is fairly straightforward in the way the program writes a single tape to a file or tabular array as determined by the Logging Service'due south configuration. In contrast, the log reading sample program is more complex considering y'all can specify that queries are applied to multiple files or tables.

---

Caution –

Log files and tables in particular can go very large. If you specify multiple logs in a single query, create queries that are very specific, or limited in the number of records to return, or both specific and express. If a large number of records are returned, the Access Manager resources limits (including those of the hosting system) may be exceeded.

---

LogReaderSample.java requires three command-line arguments which are used to authenticate with the Access Manager server. If you specify a log proper name, so the sample becomes a single-log reading application. If y'all don't specify a log name, reading from multiple logs is allowed. Reading from multiple logs does not preclude reading from a single log. Reading from multiple logs is useful when the exact log names available are unknown. The log reading sample is also very interactive. The following control-line case uses the LogReaderSample script:

./RunLogReader -o dc=iplanet,dc=com -u amadmin -p mypassword

In LogReaderSample.java, the command-line arguments are read. The following arguments are used to obtain the SSOToken that is specified in invoking the diverse LogReader.read() methods:

-o

organization name

-u

userID

-p

userID password

The LDAP login utility ldapLogin() is provided in a dissever file, LogSampleUtils.java.

Next, the Logging Service configuration is read to decide, for example, whether file or database logging is specified and which log fields are logged.

manager.readConfiguration();      String logStorageType = manager.getProperty(LogConstants.BACKEND);

Depending on whether Admission Director Logging Service is logging to a file or to a database, when the LogReader.getSize() method is invoked on a particular log name, LogReader.getSizeUnits() will return either LogConstants.NUM_BYTES or LogConstants.NUM_RECORDS. For case:

          i3 = LogReader.getSizeUnits();        

The LogConstants.LOG_FIELDS belongings specifies which log fields have been specified for inclusion in the log record. For example:

String selFldsStr = managing director.getProperty(LogConstants.LOG_FIELDS);

The time and Data fields are mandatory, thus they do not announced in the Logging Service list. They must be explicitly added to the Prepare of Fields to Recollect.

          StringTokenizer stoken = new StringTokenizer(selFldsStr, ", "); 					String [] sFields = new String[stoken.countTokens() + 3];      					Set allFields = new HashSet();  					allFields.add("time"); 					allFields.add("data");

To get the Prepare of Log Names Bachelor to read and their sizes:

          Set filesThereAre = LogReader.getLogNames(); for (Iterator it=filesThereAre.iterator(); it.hasNext(); ) { 	Cord fileName = (Cord)information technology.next(); 	long li = 0; 	endeavor { 		li = LogReader.getSize(fileName); 	} catch (Exception ex) {              		System.out.println("got exception on file " + 				fileName + ". " + ex.getMessage()); 	}          	System.out.println (fileOrTable + " " + (i2++) + 		" = " + fileName + " contains " + li + " " + 		sizeUnit + ".");      }

LogReaderSample.coffee allows you lot to select reads on a single or multiple logs. If a log proper noun was specified on the command line with the -north option, and so you can select from amid the post-obit types of reads:

            i. read all records 			2. specify logType 			iii. specify logType and timeStamp 			iv. specify logType and logQuery 			five. specify logType, timeStamp, and logQuery 			6. specify logQuery

If no log proper name was specified on the command line, and you select unmarried log to read, you lot may select from only a listing of pre—configured reports:

            Single (s) or multiple (m) file/table read: [s] 			What type of audit report to generate: 			 one. all records from file/table 			 2. authentication successes 			 three. authentication failures 			 4. login/logout activity 			 five. policy allows 			 six. policy denies 			 7. amAdmin CLI activity 			 8. amAdmin console activeness 			 9. Federation admission 			x. Federation errors 			11. Liberty access 			12. Freedom errors 			13. SAML access 			14. SAML error           					enter type [1..14]:          

If you want to read from a selected unmarried log, merely specify the logQuery settings, do non use the -n command-line option. Select multiple log read, and then select the unmarried log from which to read:

            Available files: 			file 0 = amAuthentication.access contains 1595 bytes. 			file i = amPolicy.admission contains 2515 bytes. 			... 			file thirteen = amAuthentication.error contains 795 bytes.  			Single (southward) or multiple (m) file/table read: [s]                          m                        Available files: 			0: amAuthentication.access			 			one: amPolicy.admission 			... 			12: amConsole.access-1					 			13: amAuthentication.error  			Enter selections (space-separated):                          0                        What type of read to use:  				one. read all records 				2. specify logQuery           					 enter type [ane or 2]:          

The following tabular array provides brief descriptions of the LogReader.read() methods.

Table 6–1 LogReader.read() Methods

read(String fileName, Object userCrdential)	Returns all of the records from the specified log, ignoring the maximum number of records specified in the Logging Service configuration.
read(Cord logName, String logType, Object userCrdential)	Specifies the log name and its suffix (type) separately, where the suffix can be access or error. All records are retrieved from the specified log.
read(String logName, Cord logType, String timeStamp, Object userCrdential)	Used when reading secure log files. The timeStamp is the suffix that appears after the file logType (access or fault). All records are retrieved from the specified log.
read(String logName, String logType, LogQuery logQuery, Object userCrdential)	Performs a query, every bit specified by the logQuery parameter. The log name and blazon (access or error) are also specified.
read(String logName, String logType, String timeStamp, LogQuery logQuery, Object userCrdential)	Corresponds to the method described above. Used in the secure logging case.
read(String logName, LogQuery logQuery, Object userCrdential)	Performs a query on the specified log.
read(Cord logName, Set fileNames, LogQuery logQuery, Object userCrdential)	Performs a query on the specified Set up of Logs.

The LogQuery, along with the QueryElements that may be specified, are constructed in the getLogQuery() routine in LogReaderSample.java.

The post-obit are brief descriptions of the LogQuery constructors.

LogQuery()

Creates a new LogQuery object with the post-obit default values:

maxRecord =  	LogQuery.MOST_RECENT_MAX_RECORDS 	  	globalOperand =  	LogQuery.MATCH_ANY_CONDITION 	     queries = nothing (QueryElement) 	     columns = nothing (columns to return) sortBy = cypher (field to sort on)

LogQuery(int max_record)

Creates a new LogQuery object with the post-obit values:

maxRecord = max_record globalOperand = 	LogQuery.MATCH_ANY_CONDITION queries = zero (QueryElement) columns = nil (columns to return) sortBy = null (field to sort on)

LogQuery(int max_Record, int matchCriteria, java.lang.String sortingBy)

Creates a new LogQuery object with the following values:

maxRecord = max_Record 	     globalOperand = matchCriteria 	     queries = zilch (QueryElement) 	     columns = null (columns to return) 	     sortBy = sortingBy (field to sort on)

The LogQuery object created with the constructors may be later modified with the following set* methods:

• setColumns(coffee.util.ArrayList columns)

• setGlobalOperand(int no)

• setMaxRecord(int value)

• setSortingField(java.lang.Cord fieldName)

A LogQuery may specify a List of QueryElements, each containing a value for a field (cavalcade) and a relationship. The following sample code queries for all successful authentications in domain dc=iplanet,dc=com, and returns the time, Data, MessageID, ContextID, LoginID, and Domain fields, sorted on the LoginID field:

ArrayList al = new ArrayList(); al.add (LogConstants.Fourth dimension); al.add together (LogConstants.Information); al.add (LogConstants.MESSAGE_ID); al.add together (LogConstants.CONTEXT_ID); al.add (LogConstants.LOGIN_ID); al.add (LogConstants.DOMAIN); LogQuery lq = new LogQuery(LogQuery.ALL_RECORDS, 		LogQuery.MATCH_ALL_CONDITIONS, 		LogConstants.LOGIN_ID);  QueryElement qe1 = new QueryElement(LogConstants.MESSAGE_ID, 		"AUTHENTICATION-105", 		QueryElement.EQ); lq.addQuery(qe1);  QueryElement qe2 = new QueryElement(LogConstants.DOMAIN, 		"dc=iplanet,dc=com", 		QueryElement.EQ); lq.addQuery(qe2);

QueryElement supports the following relationships:

QueryElement.GT

Greater than

QueryElement.LT

Less than

QueryElement.EQ

Equal to

QueryElement.NE

Not equal to

QueryElement.GE

Greater than or equal to

QueryElement.LE

Less than or equal to

QueryElement.CN

Contains

QueryElement.SW

Starts with

QueryElement.EW

Ends with

In the case, assuming that dc=iplanet,dc=com is the root domain, changing the qe2relationship field to QueryElement.EW (Ends with) or QueryElement.CN (Contains) changes the query to include all successful authentications in all domains. To read the instance query from the amAuthentication.admission log, assuming the SSOToken is in ssoToken:

          Cord[][] result = new String[one][1];     consequence = read("amAuthentication.access", lq, ssoToken);

The first tape (row 0) contains the field and column names. Encounter the printResults() method in LogReaderSample.java for a sample display routine.

chickbuthationd.blogspot.com

Source: https://docs.oracle.com/cd/E19462-01/819-4675/gbdcu/index.html

Share this post