A customer-inspired performance-monitoring pattern with OCI Application Performance Monitoring

This new monitoring paradigm grew out of a customer request to test the Oracle Fusion REST APIs that their business integrations depend on, with synthetic monitoring as the next step. It offers a way to check API availability and expected responses even when integration traffic is quiet.

That request points to a useful design shift for Fusion monitoring: organize checks around business API capabilities. An item lookup, an inventory query, or a work-order request gives availability a meaning that an integration team can act on. Each check combines an authenticated request, a small set of known test records, and explicit response assertions.

The customer request supplied the starting point, but the pattern applies more broadly. Fusion customers can choose the API capabilities on which their own processes depend, then check those capabilities from the locations and at the frequency that matter to them. In a broader Fusion observability approach, these outside-in API checks belong in the Performance pillar: they measure whether a selected business API responds correctly and how long the synthetic operation takes. Real User Monitoring for Oracle Fusion SaaS Cloud Apps examines the complementary experience of people using the Fusion interface, while ESS log collection and ESS, audit, and security dashboards show other operational and security signals. This pillar framing describes how to organize the monitoring design; it is not an OCI product category.

This post extends Fusion monitoring beyond native metrics by asking whether a selected, authenticated business API returns the expected data within an acceptable time from a chosen vantage point. Using OCI Application Performance Monitoring (APM) Availability Monitoring, we show how to create Scripted REST checks, configure where and when they run, and visualize availability, execution time, and failure trends. Items, inventories, work centers, and work orders illustrate scoped GET checks; a read-only batch shows how to test POST without changing business records. Together, these examples give Fusion teams a pattern they can adapt to their own critical integrations while keeping customer-specific workflows and records private.

Reference architecture: Fusion business APIs and APM Availability Monitoring

In OCI APM, these synthetic checks are configured under Availability Monitoring. APM schedules the monitor, a selected execution location sends requests to Fusion, and the results return to APM for analysis. Requests originate at the vantage point; this outside-in design needs no monitoring agent inside Fusion. The architecture connects each business API capability to a repeatable availability check. Use Availability Monitoring

Figure 1. Reference architecture for Fusion Business API endpoint monitoring via OCI APM.

The components have distinct responsibilities:

For a scripted monitor, the script defines what to test; the monitor defines how and where it runs. For example, an items script contains the lookup and assertions, while a monitor named fusion_items supplies the pod, test values, schedule, and locations.

Choose the monitor type for the test

Availability Monitoring provides several monitor types, each suited to a different kind of test:

Oracle’s monitor type and configuration documentation describes the settings for each type. SQL monitoring requires an accessible database; a Fusion REST endpoint does not provide that database connection.

We use Scripted REST because this API testing use case needs queries built from test values, validation of returned JSON and batch parts, and timing around named operations. JavaScript gives the check control over those requests and assertions. The REST monitor also supports POST; the reason for choosing Scripted REST here is the programmable test logic. Browser monitors suit checks of the Fusion user interface.

What is a vantage point?

A vantage point is where APM runs a monitor and sends its requests. Oracle provides public locations in OCI data centers and in external data centers; the locations labeled External are part of that public offering. You can also run monitors from locations you operate in your tenancy or on a VM in another network. Oracle APM vantage-point overview

This exercise uses selected public vantage points for a reachable Fusion pod. Choose additional locations based on the network path the integration uses; each location contributes its own execution result.

At each selected vantage point, the script builds the request, authenticates with the monitoring account, and checks the returned data. A failed assertion fails that execution. Comparing the same check across locations helps distinguish a widespread problem from one concentrated along a particular access path.

Create a script for an API check

Open Observability & Management → Application Performance Monitoring → Availability Monitoring → Scripts, select the compartment and APM domain, and choose Create Script. Upload a JavaScript .js file. After saving the script, create a Scripted REST monitor that references it. Oracle documents postman-request for HTTP calls and ORAP placeholders for parameters, including secret values. Create an APM script

This compact items example shows the pattern. Set BASE_URL to the HTTPS origin, BASIC_AUTH_B64 to Base64 of the UTF-8 string username:password with Value is secret enabled, and ITEM_QUERY to an organization-and-item-scoped query, such as OrganizationCode='DEMO_ORG' AND ItemNumber='DEMO_ITEM', using your own test values. Base64 is reversible encoding, so treat that authentication value as a credential. The query is supplied through qs so the request library URL-encodes it.

const request = require("postman-request");
const baseUrl = "<ORAP><ON>BASE_URL</ON><OV>SET_IN_APM</OV><OS>false</OS></ORAP>";
const basicAuth = "<ORAP><ON>BASIC_AUTH_B64</ON><OV>SET_IN_APM</OV><OS>true</OS></ORAP>";
const itemQuery = "<ORAP><ON>ITEM_QUERY</ON><OV>SET_IN_APM</OV><OS>true</OS></ORAP>";

if ([baseUrl, basicAuth, itemQuery].some(v => !v || /SET_IN_APM|<ORAP>/.test(v))) {
  throw new Error("CONFIG: set the script parameters");
}
if (!/^https:\/\/[a-z0-9.-]+$/i.test(baseUrl)) {
  throw new Error("CONFIG: BASE_URL must be an HTTPS origin without a trailing slash");
}

oraSynCustomMarker("item_lookup", "startTime");
request({
  method: "GET",
  url: baseUrl + "/fscmRestApi/resources/11.13.18.05/itemsV2",
  qs: { q: itemQuery, fields: "ItemNumber,ItemId", onlyData: true, limit: 1 },
  headers: {
    Authorization: "Basic " + basicAuth,
    Accept: "application/json",
    "REST-Framework-Version": "4"
  },
  strictSSL: true, followRedirect: false, followAllRedirects: false,
  timeout: 20000
}, (error, response, body) => {
  if (error) throw new Error("REQUEST: item_lookup");
  if (!response || response.statusCode !== 200) throw new Error("HTTP: item_lookup");
  let data;
  try { data = JSON.parse(body); }
  catch (_) { throw new Error("JSON: item_lookup"); }
  const item = data && Array.isArray(data.items) && data.items[0];
  if (!item || item.ItemNumber == null || item.ItemId == null) {
    throw new Error("ASSERT: item_lookup expected item fields");
  }
  oraSynCustomMarker("item_lookup", "endTime");
  console.log("PASS: item_lookup");
});

This is an illustrative single-request check, verified locally with mocked responses; it has not been executed as a hosted monitor. It checks status, JSON, and item-field presence. The fuller scripts add stricter input validation, response-size and pagination controls, and sanitized diagnostic detail.

The paired literal markers measure the successful item_lookup step, including response validation. Failed checks throw a short diagnostic without printing credentials or returned records. The example uses a direct ITEM_QUERY parameter for readability; the fuller monitors build their queries from the fixture supplied in FIXTURE_B64.

What data does APM collect?

APM publishes availability and monitor execution-time metrics, with dimensions that include the monitor and vantage point. These support comparisons across locations and over time. APM metric definitions

The scripts add context through named timing markers and sanitized output:

• Availability: whether the execution satisfied the configured checks. A failure can reflect connectivity, authentication, response validation, or a fixture that needs attention.
• Execution time: how long the complete monitor took, plus custom durations around named script steps. A GET step can include pagination and validation; the batch marker measures the complete batch check.
• Response evidence: whether the expected HTTP status, JSON structure, selected fields, and applicable identity checks passed. Success output includes bounded row and request counts.
• Failure context: the operation that failed and a diagnostic such as a missing parameter, an HTTP error, or a transport error. Credentials and response bodies are excluded from the script’s own logs.

These signals help direct investigation. A missing fixture value calls for a configuration check. An HTTP 401 calls for investigation of authentication and access. Slower results concentrated at one vantage point suggest comparing that location’s access path with the others before attributing the delay to Fusion.

Custom assertions may appear under APM’s Unknown Error category, so inspect the script error message as well. A completed timing marker may also be absent when the execution fails early.

Monitor the business contexts

For an integration, availability includes being able to authenticate, access the intended resource, and receive a usable response. A successful HTTP connection is one part of that contract. The script also needs to check the response against a defined expectation.

A fixture supplies that context: a known item and organization, an existing work center, or a work order selected for testing. The script uses those values to make a focused request and evaluate the result. Small page limits and a request timeout keep each execution bounded.

These resource paths sit below /fscmRestApi/resources/11.13.18.05. They are representative checks, rather than complete request definitions; each needs an appropriate query and fixture for the target pod.

The assertions determine what a green result means. In this implementation, the GET checks generally validate HTTP status, JSON content, nonempty collections, and selected field presence; they do not all compare every returned value with the fixture. The batch check additionally verifies the requested record identities. Inventory quantities and completion of a business process are outside these checks.

Exercise POST with a read-only batch

POST is useful here because Oracle’s batch interface accepts an envelope containing multiple operations, including reads. Our example posts to /fscmRestApi/resources/11.13.18.05/ with the media type application/vnd.oracle.adf.batch+json. Both enclosed operations use get. Oracle batch actions

The abbreviated body below illustrates the idea. The paths contain explanatory placeholders; the actual script constructs and URL-encodes queries from private fixture values.

{
  "parts": [
    {
      "id": "item_lookup",
      "path": "/itemsV2?<scoped item query>",
      "operation": "get"
    },
    {
      "id": "work_order",
      "path": "/workOrders?<known work-order query>",
      "operation": "get"
    }
  ]
}

A Scripted REST monitor can send that batch request with JavaScript. The example below takes BASE_URL as the HTTPS origin, BASIC_AUTH_B64 as the secret Base64 value of username:password, and FIXTURE_B64 as a secret Base64-encoded JSON object containing organization_code, item_number, and work_order_id. Supply existing test records when setting the monitor parameters. The script fixes both batch operations to get and URL-encodes their scoped queries.

const request = require("postman-request");
const baseUrl = "<ORAP><ON>BASE_URL</ON><OV>SET_IN_APM</OV><OS>false</OS></ORAP>";
const basicAuth = "<ORAP><ON>BASIC_AUTH_B64</ON><OV>SET_IN_APM</OV><OS>true</OS></ORAP>";
const fixtureB64 = "<ORAP><ON>FIXTURE_B64</ON><OV>SET_IN_APM</OV><OS>true</OS></ORAP>";
const apiPath = "/fscmRestApi/resources/11.13.18.05";
const mediaType = "application/vnd.oracle.adf.batch+json";

if ([baseUrl, basicAuth, fixtureB64].some(v => !v || /SET_IN_APM|<ORAP>/.test(v)) ||
    !/^https:\/\/[a-z0-9.-]+$/i.test(baseUrl)) {
  throw new Error("CONFIG: set valid script parameters");
}
let fixture;
try { fixture = JSON.parse(Buffer.from(fixtureB64, "base64").toString("utf8")); }
catch (_) { throw new Error("CONFIG: FIXTURE_B64"); }
function safeText(value) {
  return typeof value === "string" && !!value.trim() && value.length <= 128 &&
    !/[\x00-\x1f\x7f]/.test(value);
}
if (!fixture || typeof fixture.organization_code !== "string" ||
    typeof fixture.item_number !== "string" ||
    !/^[1-9][0-9]{0,17}$/.test(String(fixture.work_order_id)) ||
    !safeText(fixture.organization_code) || !safeText(fixture.item_number)) {
  throw new Error("CONFIG: expected item, organization, and work order");
}
function quoted(value) { return "'" + value.replace(/'/g, "''") + "'"; }
function path(resource, query, fields) {
  return "/" + resource + "?q=" + encodeURIComponent(query) +
    "&fields=" + encodeURIComponent(fields) + "&limit=1&onlyData=true";
}
const parts = [
  { id: "item_lookup", operation: "get",
    path: path("itemsV2", "OrganizationCode=" + quoted(fixture.organization_code) +
      " AND ItemNumber=" + quoted(fixture.item_number), "ItemId,ItemNumber,OrganizationCode") },
  { id: "work_order", operation: "get",
    path: path("workOrders", "WorkOrderId=" + fixture.work_order_id,
      "WorkOrderId,WorkOrderNumber") }
];

oraSynCustomMarker("batch_readonly", "startTime");
request({
  method: "POST", url: baseUrl + apiPath + "/",
  body: JSON.stringify({ parts }),
  headers: { Authorization: "Basic " + basicAuth, Accept: mediaType,
    "Content-Type": mediaType, "REST-Framework-Version": "4" },
  strictSSL: true, followRedirect: false, followAllRedirects: false,
  timeout: 20000
}, (error, response, body) => {
  if (error) throw new Error("REQUEST: batch_readonly");
  if (!response || response.statusCode !== 200) throw new Error("HTTP: batch_readonly");
  const contentType = response.headers && response.headers["content-type"];
  if (typeof contentType !== "string" ||
      !/^application\/vnd\.oracle\.adf\.batch\+json(?:\s*;.*)?$/i.test(contentType)) {
    throw new Error("CONTENT_TYPE: batch_readonly");
  }
  if ((typeof body !== "string" && !Buffer.isBuffer(body)) ||
      Buffer.byteLength(body) > 2 * 1024 * 1024) {
    throw new Error("BODY: batch_readonly");
  }
  let result;
  try { result = JSON.parse(body.toString("utf8")); }
  catch (_) { throw new Error("JSON: batch_readonly"); }
  if (!result || !Array.isArray(result.parts) || result.parts.length !== 2) {
    throw new Error("BATCH: expected two parts");
  }
  const byId = Object.create(null);
  for (const part of result.parts) {
    if (!part || !["item_lookup", "work_order"].includes(part.id) ||
        byId[part.id] || part.operation !== "get" || !part.payload ||
        !Array.isArray(part.payload.items) || part.payload.items.length !== 1) {
      throw new Error("BATCH: missing or invalid read part");
    }
    byId[part.id] = part.payload.items[0];
  }
  const item = byId.item_lookup;
  const order = byId.work_order;
  if (!item || item.ItemId == null || item.ItemNumber !== fixture.item_number ||
      item.OrganizationCode !== fixture.organization_code || !order ||
      String(order.WorkOrderId) !== String(fixture.work_order_id) ||
      !order.WorkOrderNumber) {
    throw new Error("ASSERT: batch_readonly identities");
  }
  oraSynCustomMarker("batch_readonly", "endTime");
  console.log("PASS: batch_readonly parts=2");
});

This compact example was verified locally with mocked responses; this exact excerpt has not been run as a hosted monitor. The fuller batch script adds stricter input, returned-path, pagination, and response-size handling. The monitoring account and fixture remain private.

The monitor validates the outer response and both returned parts, including their identities. HTTP 200 alone is insufficient: a missing, duplicate, erroneous, or mismatched part fails the check. The implementation fixes the two read operations in code rather than accepting arbitrary batch actions through parameters.

This exercises POST transport, authentication, batch processing, and result validation without creating or updating business records. It provides a useful additional monitoring pattern alongside the individual GET checks. Testing a business write would require its own transaction assertions and repeatable data lifecycle.

Monitor Fusion API endpoints in APM

Start with an APM domain, a monitoring account that can access the selected Fusion records, and a vantage point that can reach the pod. Prepare one stable fixture and confirm the request locally before scheduling it.

1. Upload and configure the script

In Observability & Management → Application Performance Monitoring → Availability Monitoring, select the compartment and APM domain, open Scripts, and choose Create Script. On Script Definition, upload one .js file for the capability you want to check. For example, the items script contains its HTTP request and assertions; the read-only batch script contains its POST and two fixed GET parts. Confirm that the uploaded content is the intended version before continuing. Create an APM script

Figure 2. Uploading the full items Scripted REST file.

The script’s ORAP placeholders appear as Script Parameters below the content. Set BASE_URL to the Fusion HTTPS origin without a trailing slash. Set BASIC_AUTH_B64 to the Base64 value of username:password and turn on Value is secret. Set FIXTURE_B64 to Base64-encoded JSON containing the selected test identifiers and mark it secret when those identifiers are private. The earlier capture below has the fixture’s secret toggle off; enable it for private identifiers before saving. Base64 is reversible encoding, so neither value is safe to publish. Keep the script’s request-size, page-count, and timeout settings bounded.

Figure 3. Configuring script parameters and secret values.

Choose Next, add tags if needed, and check the Summary page before selecting Create. Verify the script name and parameter names; a summary only confirms what will be saved, not that the API request has succeeded. The capture below shows the equivalent summary for the read-only batch script.

Figure 4. Reviewing the read-only batch script summary.

2. Create and configure the monitor

From the saved script’s page, choose Create Monitor, or open Monitors → Create Monitor in the same APM domain. Enter a distinctive name, select Scripted REST, and choose the saved .js script. The script parameters are available on Monitor Definition; verify their values for this environment and keep BASIC_AUTH_B64 secret.

Figure 5. Historical monitor definition with an empty monitor-level Base URL.

Enter the Fusion HTTPS origin, without a trailing slash, in both places: the monitor-level Base URL field and the script parameter BASE_URL. The monitor form in this setup requires its own Base URL even though the script builds complete request URLs from its parameter. Oracle’s documentation labels the monitor-level field optional, so check the behavior of the APM form you use; the earlier empty-field capture above is no longer the configuration to follow here. Scroll down to Authentication and leave its type at None: this script supplies the Basic Authorization header itself. Under Response Validations, select response code 200; the JavaScript also checks the status and the returned data. Leave Override DNS off unless the network design specifically requires it. Oracle monitor configuration

Figure 6. Configuring authentication, response validation, and DNS options.

3. Set the execution locations and cadence

On Run Settings, select the vantage points that represent the integration’s access path. For an initial diagnostic, choose one location and Run once; once that execution succeeds, expand to the intended locations and select Interval for continuous monitoring. Set the monitor timeout long enough for the script to finish; the captured setup used three minutes, while each request in the example has its own 20-second timeout. Disable Enable Retry for the first diagnostic so its result is easier to interpret, then decide whether retries fit the operational policy. Figure 7 captures a later choice of four public vantage points, Run once, and retry enabled.

Figure 7. Selecting vantage points and run settings.

For the current setup, leave Enable Network Collection unchecked even though both Base URL fields are populated. The image above shows the same unchecked option in the earlier configuration with an empty monitor-level Base URL. Network Collection is a separate measurement option; leaving it unchecked here does not imply a general incompatibility with Fusion. Oracle documents that the option is shown for Scripted REST when a monitor-level Base URL is specified. Run settings and Network Collection

4. Define availability and review the summary

On Availability Configuration (Optional), enable the calculation if you want the dashboard’s availability widget to classify each interval. The captured example allows 0 maximum failures and requires 1 minimum run per interval. With those thresholds, any failed run makes an interval with enough runs unavailable; too few runs makes it unknown and excludes it from the availability calculation. Choose thresholds appropriate for the final schedule and number of vantage points. Availability configuration

Figure 8. Setting availability thresholds for monitor runs.

Choose Next, add optional tags, and inspect Summary. Confirm that the monitor-level Base URL and the script parameter BASE_URL both show the intended Fusion origin. Also review the script name, authentication type, response validation, vantage points, cadence, retry, Network Collection, and availability thresholds before clicking Create. The summary screenshot is a historical configuration review, not proof of a completed run or an example of the new Base URL setting.

Figure 9. Reviewing the historical monitor summary before creation.

Inspect the newest execution in History before choosing a recurring interval. Verify the timestamp, vantage point, assertions, and duration. Then add the other selected capabilities, each with its own fixture, and expand the locations according to the integration’s access requirements.

Development evidence included a successful hosted items check and local validation of the other selected GET patterns. The exact read-only batch script passed locally against Fusion. The later dashboard screenshots also show recorded successes and failures for the hosted batch monitor. Those aggregate counters do not identify the executed script revision or establish which response assertions passed; use individual run output to confirm that baseline.

Create monitoring widgets using APM monitor KPIs

Each Availability Monitoring execution produces signals that can answer three operational questions: Is the Fusion endpoint available? How long does the scripted check take? Are failures recurring or concentrated at one vantage point? Build widgets around those questions so the dashboard shows the health of the business API checks, not merely a collection of charts. The supplied test dashboard includes historical series beyond the five examples in this article; its successes and failures are not a production SLA.

Figure 10. Viewing availability and run results in the APM dashboard.

Start with a small KPI set, then choose a widget that makes each signal easy to interpret. The metrics below come from the oracle_apm_synthetics namespace. They measure synthetic monitor runs and requests, not the volume or end-to-end success of the customer’s integration traffic. APM metric definitions

For count metrics, sum within a time bucket. For execution times, start with a mean and keep the native millisecond unit clear; compare failure counts before treating a sudden shorter run as an improvement. Choose a bucket that fits the monitor’s schedule. APM metric names, units, and dimensions

Create the widgets. A Metric line chart is one way to show execution time or availability over time; use a table or count chart when it communicates outcomes more clearly. The following steps use the APM dashboard editor:

1. Open Observability & Management → Application Performance Monitoring → Dashboards. Create a dashboard or open a custom dashboard and choose Actions → Edit. To customize an Oracle-defined dashboard, duplicate it first. APM dashboard customization
2. On the Widgets tab, choose a visualization for one KPI. For example, search for Metric line chart and add it to the canvas for execution time.
3. Open Edit widgets, select that widget, and use the pencil icons beside Configured widget inputs. Set Namespace to oracle_apm_synthetics and Metric name to MonitorExecutionTime; verify that the data compartment contains the intended APM domain.
4. Label it Fusion monitor execution time, select a time range, and choose Save changes. Add separate widgets for availability and failure counts using the metrics in the table. Confirm that each series represents the intended monitor and location. Widget input configuration

An APM domain filter elsewhere on a dashboard does not automatically scope every generic metric widget. Check its input links. When you need explicit domain filtering and separate vantage-point series that the widget’s exposed inputs do not provide, use Widgets → + → Create query-based widget…, select Monitoring as the data source and oracle_apm_synthetics as the namespace, then enter a query such as:

MonitorExecutionTime[10m]{ResourceId="<APM_DOMAIN_OCID>",MonitorName="fusion_items"}.groupBy(MonitorName,VantagePoint).mean()

Replace the placeholders with your domain and monitor. This illustrative query uses a ten-minute aggregation interval and a mean for each monitor/location pair; the dashboard time range is a separate choice. Click Run, select a Line chart visualization, save the widget, and save the dashboard. Query-based metric widgets, Monitoring query syntax

Use MonitorName for a capability overview and VantagePoint to compare locations. ResourceId identifies the APM domain; MonitorId identifies an individual monitor. Prefer monitor names in public chart legends because Target can expose pod URLs. In Figure 11, MonitorExecutionTime measures the complete monitor run; the separate Latency metric measures network packet round-trip time, not the duration of a Fusion API operation. APM metric dimensions

Figure 11. Charting synthetic requests, failures, and monitor execution time.

Use RequestCount to understand synthetic request activity, not the customer’s integration traffic volume. A shorter execution time can also accompany a request that fails early, so compare timings with success and failure signals. Gaps are missing observations, not evidence of zero latency or perfect availability. Select the same monitor and time window in History to investigate a change. APM synthetic metric definitions

Conclusion: make Fusion API health visible

This customer-inspired pattern turns an integration requirement into a repeatable check at the Fusion SaaS API boundary. A Scripted REST monitor asks a specific business question—can this account retrieve the expected item, inventory reference, work center, or work order?—from a chosen vantage point. A read-only batch adds coverage of POST transport without changing business records. The resulting availability, execution-time, and failure signals remain useful even when customer integration traffic is quiet.

The broader value for Fusion customers is a shared view of which capability is affected, when it changed, and from where the failure was observed. A dashboard can reveal a trend; the individual monitor run provides the request and assertion context needed to investigate it. As a Performance-pillar signal, this outside-in view complements real user experience monitoring, ESS process logs, integration logs, and security/audit analysis. A passing synthetic read does not prove that every business transaction or customer integration succeeded.

The five examples are a starting template, not a prescribed customer workload. Start with a few critical endpoints and stable, owned test records. Assign an owner to each fixture and monitor, validate the script from the intended vantage points, then schedule and review the checks as the integration evolves. That makes API availability an operational signal tied to business capabilities rather than an isolated pod status.

References

• Oracle APM overview: public locations in OCI and external data centers
• Set Up Availability Monitoring: public and private vantage-point options
• Use Dedicated Vantage Points
• Use On-premise Vantage Points
• Download a Worker: monitor types supported by each on-premise worker
• APM Availability Monitoring metrics and dimensions
• Create a query-based widget using metrics

Related Fusion observability posts by Royce Fu and coauthors:

• Real User Monitoring for Oracle Fusion SaaS Cloud Apps using OCI APM Browser Agent — browser-side user experience and performance.
• Fusion Apps Observability by Collecting Enterprise Scheduler ESS Logs Using OCI Logging Analytics REST API Ingestion — scheduled-process logs and operational visibility.
• Gaining Insights from Fusion ESS, Audit and Security Dashboards with AI Augmentation Using LoganAI – Part 3 — ESS, audit, and security analysis.

Given the complexity of managing an Exadata system, observability and monitoring are essential to achieving operational readiness. Comprehensive observability enables administrators to gain deep insights into system performance, detect anomalies, and address potential issues before they escalate into significant problems. To achieve comprehensive observability and management of Exadata systems, we recommend utilizing OCI Observability and Management (O&M) services such as Management Agent Cloud Service, Database Management Service, Ops Insights, and Logging Analytics. These services enable the collection and ingestion of Exadata system metrics and log data across various environments, including OCI, Cloud@Customer, and on-premises deployments.

This article focuses on OCI Log Analytics for Oracle Database@AWS Exadata Database Service on Dedicated Infrastructure (ExaDB-D). The same ZIP-based Management Agent and log-collection model also applies to Exadata Database Service on Cloud@Customer (ExaC@C). The walkthrough covers Management Agent installation, database and Grid Infrastructure (GI) log access, entity and source associations, Log Explorer verification, dashboards, and investigation queries. It builds on Maximizing Visibility: Collecting and Analyzing Logs from Oracle Databases using OCI Logging Analytics.

Oracle’s multicloud database monitoring strategy

Oracle’s multicloud database monitoring strategy combines three complementary OCI Observability and Management services. Database Management provides fleet monitoring, database and infrastructure metrics, performance management, and administration. Ops Insights adds capacity planning, SQL insights, machine-learning-driven diagnostics, and fleet-level trends. Log Analytics centralizes logs from cloud and on-premises environments and applies Oracle-aware parsing, interactive analytics, dashboards, and AI-assisted investigation. Together, these services let operations teams move from a fleet symptom to database performance, capacity, and log evidence without treating any single signal as the complete answer. OCI monitoring for Oracle Database@AWS, Database Management, Ops Insights

Figure 1. Oracle’s database management strategy combines Database Management, Ops Insights, and Log Analytics across public-cloud, hybrid-cloud, and multicloud database deployments.

Why database and GI logs become difficult to use across a fleet

Oracle Database@AWS publishes useful performance and infrastructure metrics to Amazon CloudWatch under the AWS/ODB namespace. The service view can expose CPU utilization, load, memory, ASM disk-group utilization, node status, allocated OCPUs, file-system utilization, active sessions, and I/O performance, depending on the resource and enabled monitoring. These metrics help operators recognize resource pressure and availability changes across Exadata VM clusters. Oracle Database@AWS CloudWatch monitoring, AWS monitoring overview

Figure 2. CloudWatch provides useful VM-cluster and database metrics. The VM-cluster name and database-node OCIDs were redacted for publication.

The CloudWatch Logs view captured for this walkthrough primarily exposes the /aws/events/odba_events log group. The example below records a FetchProtectedDatabaseConfiguration operation as an audit-style service event. AWS documents Oracle Database@AWS events as structured messages about resource lifecycle changes, including Exadata infrastructure, VM cluster, CDB, and PDB events. CloudTrail separately records Oracle Database@AWS control-plane API calls. These streams are valuable for tracking provisioning, configuration changes, and administrative activity, but they are different from the diagnostic files generated inside the database servers. Oracle Database@AWS events, Oracle Database@AWS CloudTrail logging

Figure 3. An audit-style Oracle Database@AWS service event in /aws/events/odba_events. Account identifiers, ARNs, OCIDs, event identifiers, the IP address, and other environment-specific values were redacted for publication.

Database alert logs, listener logs, ASM logs, and Clusterware logs are produced on the Exadata database VMs. Without centralized collection, a DBA investigating an ORA-07445, ORA-00600, listener failure, or Clusterware event must connect to the relevant database nodes, traverse the database and Grid Infrastructure diagnostic directory structures, identify the correct instance and time range, and then correlate records across nodes. That process becomes slower and less consistent as the number of VM clusters grows.

Exadata diagnostics are also distributed across components and VM-local directories. A RAC database has instance-specific diagnostics; listeners, ASM, and Clusterware have their own files and ownership. Troubleshooting therefore requires both access to the right files and enough context to know which component produced each record.

As the fleet grows, several practical issues accumulate. Nodes can have different database homes and ADR paths. File access differs between the database owner, GI owner, and collection account. Rotation can remove incident evidence before someone retrieves it. A newly added VM can be missing a source association even though the rest of the cluster is collecting successfully.

Use a small collection inventory to turn those issues into explicit configuration:

This article collects from customer-accessible Exadata database VMs in Oracle Database@AWS and ExaC@C. Oracle-managed physical hosts and Exadata storage cells have different operating responsibilities. The presence of an Exadata parser in a catalog does not provide customer access to the infrastructure that produces those logs. ExaDB-D responsibility model, ExaC@C architecture

The value of OCI Log Analytics for multicloud databases

The cloud-native operating model is to deploy an OCI Management Agent with the Log Analytics plug-in on every Exadata database VM whose local logs must be monitored. Each agent reads the authorized database and GI files and sends their records to OCI Log Analytics, where the operations team can search, parse, correlate, visualize, and report on them as a fleet. This removes the need to begin every investigation by signing in to each node and manually searching separate diagnostic trees.

For a fleet of ExaDB-D or ExaC@C VM clusters, a useful log service must do more than retain files. Operators need to search the same error across instances, distinguish recurring messages from new patterns, and follow an incident from a database symptom into its listener, GI, and guest-host evidence.

OCI Log Analytics provides log collection, parsing, enrichment, search, and visualization in one service. Its catalog includes more than 250 prebuilt parsers across Oracle and other technologies. That lets a database operations team begin with Oracle-defined content and extend it when a diagnostic format needs custom handling. OCI Log Analytics capabilities, OCI Log Analytics for Oracle Database@AWS

The operational value is a common place to answer concrete questions:

• Scope: Is an error isolated to an instance, repeated on both RAC nodes, or appearing across several VM clusters?
• Sequence: Did a listener failure follow a database event, or did both coincide with a GI event?
• Change: Which message patterns appeared during the incident that were absent in the earlier comparison window?
• Evidence: Which exact records support the working explanation and the handoff to the next operations team?

Build the operating model around OCI Log Analytics

Collect at the VM, investigate across the fleet

Deploy a standalone Management Agent with the Log Analytics plug-in on each guest VM whose local logs you need. Represent the emitting components with Log Analytics entities, associate their sources, and send records to the intended log group. This preserves the instance and node context needed for fleet analysis. Continuous collection workflow

Figure 4. Proposed collection architecture. Each VM contributes its accessible logs to OCI Log Analytics. The regional connectivity path must be validated for the target deployment.

Keep three configuration concepts distinct. A source defines how records are collected and processed; a parser extracts fields; an entity identifies the component that generated them. A log group is the destination for collected records, governed through OCI IAM and compartments. Log Analytics concepts

Create a naming convention such as aws-prod-cluster01-vm01-db01 for an instance entity and record its relationship to the host and VM cluster. Maintain an expected inventory of VMs, instances, and source associations. Where operators need queryable AWS account or cluster dimensions, configure and verify the relevant enrichment; OCI resource tags should not be assumed to appear automatically as fields on every log record.

Establish the AWS-to-OCI collection path

AWS documents that an ODB network corresponds to a VCN in the linked OCI tenancy. Oracle’s Database@AWS security prerequisites illustrate a Service Gateway route to OCI Vault, while the OCI service matrix lists Management Agent, LogAnalytics, Monitoring, and Monitoring Ingestion as supported through Service Gateway. These are the basis for the proposed private service path in Figure 4. They do not replace an end-to-end check of Log Analytics collection from the target guest VM. AWS ODB networking, Oracle Database@AWS network prerequisites, Service Gateway support matrix

Before installing, have the network owner verify the linked OCI region, existing gateway and route configuration, DNS resolution, and permitted HTTPS egress. Relevant endpoint patterns include:

Use the full current prerequisites for the deployment; this table is not an exhaustive allowlist. Validate TLS with certificate verification enabled. A successful connection test establishes reachability, while agent registration and a newly ingested record establish that the collection workflow works. Agent connectivity diagnostics, Log Analytics agent endpoints

Turn records into explanations and repeatable investigations

Log Explorer supports interactive investigation as well as saved queries. Cluster analysis groups recurring message patterns; Link provides ways to analyze related records. Use these capabilities to narrow an incident before reading the underlying messages in detail. Advanced analytics

LoganAI adds generated explanations, summaries, and context-aware follow-up questions for selected logs and visualizations. For example, an operator can ask for a summary of database errors in a time window, then investigate the suggested next question against GI and host evidence. This assists RCA; the operator still verifies the conclusion against the records. Enabling LoganAI requires the documented Generative AI region, IAM, and service prerequisites. LoganAI, Enablement prerequisites

For a broader application-to-database view, OCI Log Analytics also accepts OpenTelemetry logs through the documented UploadOtlpLogs API using OTLP JSON. This is an optional ingestion path for application context; it is separate from the guest-VM file collection configured here. OpenTelemetry log ingestion

Install the Management Agent and configure ExaDB-D and ExaC@C log collection

The procedure below follows Oracle’s public Exadata ZIP installation guidance, which explicitly prohibits RPM installation on Exadata and specifies /u02 for Exadata Cloud in OCI and Cloud@Customer. It applies directly to ExaDB-D in OCI and ExaC@C; this article uses an Oracle Database@AWS ExaDB-D guest VM as the worked example. Before applying the AWS adaptation, confirm the applicable image and lifecycle requirements in MOS 3015115.1, OCI: Observability & Management Support For Exadata Cloud, with Oracle Support when needed. The public guide does not explicitly name AWS in its /u02 statement, and connectivity differs among AWS, OCI, and the customer data center. Exadata installation guidance

Step 1: Establish the OCI destination and permissions

Enable Log Analytics in the chosen OCI region and create the destination log group. Record the region, agent compartment, log-group compartment, and log-group OCID. Keep human administration separate from the identity used by the running collectors.

For example, delegate agent administration to an OCI group, substituting your identity-domain and compartment names:

Allow group <domain>/LogAnalyticsAdmins to manage management-agents in compartment <agent_compartment>
Allow group <domain>/LogAnalyticsAdmins to manage management-agent-install-keys in compartment <agent_compartment>
Allow group <domain>/LogAnalyticsAdmins to read metrics in compartment <agent_compartment>

These statements cover agent administration. For Log Analytics configuration, use an existing authorized role or add the following service permissions, applying resource access to each compartment containing entities or log groups:

Allow group <domain>/LogAnalyticsAdmins to use loganalytics-features-family in tenancy
Allow group <domain>/LogAnalyticsAdmins to use loganalytics-resources-family in compartment <log_configuration_compartment>
Allow group <domain>/LogAnalyticsAdmins to manage management-dashboard-family in compartment <dashboard_compartment>
Allow group <domain>/LogAnalyticsAdmins to read compartments in tenancy
Allow service loganalytics to read loganalytics-features-family in tenancy

Initial service enablement requires the documented administrator access; the onboarding workflow may have already created the service policy. Features-family permissions are tenancy-wide, while resource-family permissions can be compartment-scoped. Log Analytics access setup

Have an authorized IAM administrator create the policies and a dynamic group for collectors in the agent compartment:

ALL {resource.type='managementagent', resource.compartment.id='<agent_compartment_ocid>'}

Grant this group permission to upload logs, emit collection metrics, and read tenancy user information. Oracle’s policy templates support selecting the destination compartments for the first two grants; READ USERS is tenancy-scoped:

Allow dynamic-group <domain>/LogAnalyticsAgents to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} in compartment <log_group_compartment>
Allow dynamic-group <domain>/LogAnalyticsAgents to use metrics in compartment <collection_metrics_compartment>
Allow dynamic-group <domain>/LogAnalyticsAgents to read users in tenancy

Match the dynamic group’s compartment to the install key’s compartment. Select the metrics compartment actually used by your collector configuration. IAM authorizes the agent’s OCI operations; filesystem permissions separately determine which local logs it can read. Continuous-collection IAM policies, compartment-scoped templates, automatic-ingestion policy example

Step 2: Verify host prerequisites

Complete the network validation in Section 3 before continuing on each VM.

Confirm sudo access, time synchronization, adequate free space on /u02, and that the host has unzip. The observed 260729.1846 installer checks JAVA_HOME first and then /usr/bin/java; on the validated guest it found Java 1.8.0_491. Before installation, verify that root can read and execute the supported Java runtime. Set JAVA_HOME only when the required Java is not available through the default path, then verify it from the root shell. Follow the prerequisites for the exact agent release. Management Agent installation

The examples use the host’s unzip utility. Confirm it is available before the installation window.

Step 3: Prepare the ZIP installation

Download the Linux ZIP and its published SHA-256 checksum from Management Agent administration. Create a time-limited install key for the planned node count and download its response-file template. Transfer these through your approved administration channel.

For a new installation, confirm that /opt/oracle already exists, then create the external-volume target and the symbolic link exactly as the documented ZIP procedure specifies:

sudo mkdir -p /u02/app/oracle/mgmt_agent
sudo ln -s /u02/app/oracle/mgmt_agent /opt/oracle/mgmt_agent

Do not run these commands over an existing agent directory or symlink; inspect an existing deployment before changing it. The installer creates and owns the default mgmt_agent operating-system user. Do not pre-create a group, user, or custom home solely for this procedure. The /u02 requirement applies to the agent installation directory, not to the account home. Management Agent installation

Step 4: Extract, configure, and install

Use the Exadata layout from the earlier A-Team procedure: create install_dir below the external /u02 agent path and extract the Linux ZIP there. Because the directory is created with sudo and is normally owned by root at this point, this example also uses sudo for extraction. This avoids depending on whether the staging user can write to the new directory.

sudo mkdir -p /u02/app/oracle/mgmt_agent/install_dir
sudo unzip /home/opc/oracle.mgmt_agent.260729.1846.Linux-x86_64.zip \
  -d /u02/app/oracle/mgmt_agent/install_dir
cd /u02/app/oracle/mgmt_agent/install_dir/mgmt_agent

The observed ZIP creates a nested mgmt_agent directory containing installer.sh; running the installer from install_dir itself returns No such file or directory. The ZIP filename and /home/opc source path reflect the observed environment. Before extracting, compare the ZIP’s sha256sum output with the checksum in Management Agent administration, as Oracle recommends.

Before opening the root shell or running the installer, edit and securely save mgmt_agent_exadbd.rsp. Use the downloaded install key in the local file, but never capture it in screenshots, tickets, or source control. The following is a screenshot-safe rendering: it retains the observed display name and plug-in options while redacting the sensitive value.

# Do not replace this marker with the actual key in screenshots or published material.
ManagementAgentInstallKey = <REDACTED-INSTALL-KEY>
AgentDisplayName = odb-aws-host-vak331

#FreeFormTags = [{"<key1>":"<value1>"}, {"<key2>":"<value2>"}]
#DefinedTags = [{"namespace1":{"<key1>":"<value1>"}}, {"namespace2":{"<key2>":"<value2>"}}]
#CredentialWalletPassword =
#Service.plugin.jm-container.download=true
#Service.plugin.appmgmt.download=true
#Service.plugin.jms.download=true
#Service.plugin.dbaas.download=true
Service.plugin.logan.download=true
#Service.plugin.mds.download=true
#Service.plugin.osmh.download=true
#Service.plugin.opsiHost.download=true
#Service.plugin.jm.download=true
#Service.plugin..startup.jersey.hotpath.1.2f97483d-2052-4ce7-b18e-59a835d599c4.plugin.download=true

Only Service.plugin.logan.download=true is enabled in this response file; the remaining shown plug-in options are commented out. Add required defined tags and any approved gateway settings from your environment. Do not paste the key into shell history. The response file contains the installation key, so retain it in the approved protected transfer location and delete or store it securely after successful registration. Oracle requires the response file and every parent directory to be accessible to the mgmt_agent OS user.

Figure 5. Screenshot-safe rendering of mgmt_agent_exadbd.rsp; the install key is redacted, while the observed agent display name and response-file options are retained.

Important — stage the response file in /tmp before running installer.sh. The installer creates mgmt_agent and then configures the agent as that user. A response file under /home/opc may be unreadable because of parent-directory permissions. Copy the completed response file to /tmp, then grant only the temporary read access required during bootstrap:

sudo cp /home/opc/mgmt_agent_exadbd.rsp /tmp/mgmt_agent_exadbd.rsp
sudo chmod 0644 /tmp/mgmt_agent_exadbd.rsp

The validated troubleshooting session used mode 777, which proved that permissions caused the failure. Do not use 777 in the published procedure: the response file contains an install key, and mode 777 lets every local user read, modify, and execute it. Mode 0644 is the minimum practical bootstrap permission before the installer creates mgmt_agent; remove the file immediately after successful configuration.

After the response file is saved and accessible to the agent user, explicitly change into the nested directory that contains installer.sh, then open a root shell:

cd /u02/app/oracle/mgmt_agent/install_dir/mgmt_agent
sudo /bin/bash

At the root-shell prompt, start the installation:

export OPT_ORACLE_SYMLINK=true
./installer.sh /tmp/mgmt_agent_exadbd.rsp

If installation succeeds but agent configuration fails

The observed installation completed the software copy and service creation, then failed configuration because the new mgmt_agent user could not read /home/opc/mgmt_agent_exadbd.rsp. This is a partial success: do not reinstall the ZIP. Review the configure log first:

The following host output was captured before the /tmp correction, when installer.sh received the response file under /home/opc. It shows the distinction between successful software installation and failed agent configuration:

Checking pre-requisites
        Checking if any previous agent service exists
        Checking if OS has systemd or initd
        Checking available disk space for agent install
        Checking if /opt/oracle/mgmt_agent directory exists
        Checking if 'mgmt_agent' user exists
        'mgmt_agent' user does not exist. Proceeding to create the user...
        User home directory is valid and writable. Proceeding with user creation...
        Creating user on Linux-based OS...
        Checking Java version
                JAVA_HOME is not set or not readable to root
                Trying default path /usr/bin/java
                Java version: 1.8.0_491 found at /usr/bin/java
        Checking agent version

Executing install
        Unpacking software zip
        Copying files to destination dir (/opt/oracle/mgmt_agent)
        Initializing software from template
        Checking if JavaScript engine is available to use
        Creating mgmt_agent daemon
        Agent Install Logs: /opt/oracle/mgmt_agent/installer-logs/installer.log.0

Agent install successful

/home/opc/mgmt_agent_exadbd.rsp file is either not present or not readable to mgmt_agent user
Please make sure that the directory and parent directories also have the read permissions to make this file readable.

You may execute the following commands to make sure mgmt_agent user can read input.rsp:
cd /tmp
sudo -u mgmt_agent cat <FULL_PATH>/input.rsp

Missing mandatory properties opts
Usage: setup.sh opts=<FULL-PATH>
Please make sure there is no space in between opts=<FULL-PATH>

Executing configure

Prerequisite check failed
Pass opts=<> to application at the startup
Additional details are available in the configure logs at /u02/app/oracle/mgmt_agent/configure-logs

Agent configure failed.
Additional logs can be found at /opt/oracle/mgmt_agent/configure-logs/configure.log.0

sudo tail -n 200 /opt/oracle/mgmt_agent/configure-logs/configure.log.0
sudo -u mgmt_agent cat /home/opc/mgmt_agent_exadbd.rsp >/dev/null

If the second command reports Permission denied, copy the response file to /tmp. Because mgmt_agent now exists, assign the file directly to that user with mode 0600, then verify access without printing the install key:

sudo cp /home/opc/mgmt_agent_exadbd.rsp /tmp/mgmt_agent_exadbd.rsp
sudo chown mgmt_agent:mgmt_agent /tmp/mgmt_agent_exadbd.rsp
sudo chmod 0600 /tmp/mgmt_agent_exadbd.rsp
sudo -u mgmt_agent cat /tmp/mgmt_agent_exadbd.rsp >/dev/null

Rerun only the configuration step. The opts= argument is mandatory and must not contain spaces around the equals sign:

sudo /opt/oracle/mgmt_agent/agent_inst/bin/setup.sh \
  opts=/tmp/mgmt_agent_exadbd.rsp

The corrected retry produced the following successful configuration output:

Executing configure
        Parsing input response file
        Validating install key
        Generating communication wallet
        Generating security artifacts
        Registering Management Agent
                Found service plugin(s): [logan]

Starting agent...
Agent started successfully

Starting plugin deployment for: [logan]
Deploying service plugin(s)...Done.
logan : Successfully deployed service plugin

Agent setup completed and the agent is running.
In the future agent can be started by directly running: sudo systemctl start mgmt_agent
Please use OCI CLI or OCI Management Agent console to validate the successful activation of your agent.

Please make sure that you delete /tmp/mgmt_agent_exadbd.rsp or store it in secure location.

On the validated guest, this retry completed install-key validation, wallet generation, registration, agent startup, and deployment of the logan plug-in. If configuration fails again, preserve /opt/oracle/mgmt_agent/configure-logs/configure.log.0 and check the exact reported cause. Common next checks are an expired or exhausted install key, incorrect gateway/proxy parameters, and DNS, TLS, or TCP 443 connectivity to the configured OCI regional endpoint. Correct the response file or connectivity issue, verify readability again as mgmt_agent, and rerun the same setup.sh opts=... command. Management Agent setup

After setup succeeds and the agent appears active in OCI, remove the temporary response file:

sudo rm -f /tmp/mgmt_agent_exadbd.rsp

Step 5: Validate the agent and plug-in

Check the process and physical location:

systemctl status mgmt_agent --no-pager
readlink -f /opt/oracle/mgmt_agent
grep -F 'packageType=ZIP' \
  /opt/oracle/mgmt_agent/agent_inst/config/security/resource/agent.package
rpm -qa | grep -i mgmt_agent || true
stat -c '%U %n' /opt/oracle/mgmt_agent
stat -c '%U %n' /opt/oracle/mgmt_agent/agent_inst

systemctl confirms the service state. readlink confirms that /opt/oracle/mgmt_agent resolves to the /u02 installation target; agent.package must report packageType=ZIP; the RPM query must produce no Management Agent package; and both ownership checks must report mgmt_agent.

Next, validate registration and communication in the Management Agent Cloud Service:

1. In the OCI Console, select the region where the Management Agent resources were created.
2. Open Observability & Management → Management Agent → Agents and select the agent compartment.
3. Search for the node-specific AgentDisplayName, such as odb-aws-host-vak331 or odb-aws-host-vak332.
4. Confirm that Availability is Active, Operating system is LINUX, and Version matches the installed ZIP build. Review Upgrade for any required update.
5. Confirm that Service plug-ins lists Log Analytics. Open the agent details and verify that the Log Analytics plug-in status is Running.
6. Repeat the check for every database node and reconcile the console list with the planned VM-cluster inventory. Management Agent console

Figure 6. Management Agent Cloud Service validation for two ExaDB-D database nodes. Both agents are active on version 260729.1846, show Up to date, and list the Log Analytics plug-in. The compartment identifier and private DNS suffixes were replaced for publication.

An Active agent establishes cloud registration and recent Management Agent communication; it does not by itself prove that database or GI logs are being ingested. Associate the intended host, database, and cluster entities and sources, then verify a recent event from each node in Log Explorer. Check collection warnings and metrics, and test rotated or newly created files as well as existing files. For selected log paths, use ACLs that grant directory traversal and file read access, including appropriate default ACLs. Log permissions

For this Log Analytics-focused article, Service.plugin.logan.download=true is the only required plug-in. Enable Database Management or Ops Insights only when those services are part of the deployment; they are not prerequisites for collecting database or GI logs in OCI Log Analytics. If the Log Analytics plug-in was not enabled in the response file, it can instead be deployed from the Management Agent page in the OCI Console. Earlier A-Team procedure

Step 6: Install and configure the agent on every database node

After the first database node completes configuration and passes the Step 5 checks, copy the same Management Agent ZIP and a protected copy of the response-file template to every remaining Exadata database node. Install a separate Management Agent locally on each node whose database or GI files must be collected. Do not copy the installed /u02/app/oracle/mgmt_agent directory from one node to another.

Before reusing the response file, open Observability & Management → Management Agent → Administration → Install keys. Confirm that the key is active and that its remaining installations can accommodate all remaining nodes. Create or download a new install key when the existing key is expired, exhausted, or its Maximum installations setting does not cover the planned node count. Install-key administration

Use the approved encrypted transfer method for the environment. For example, from the node or staging host that currently holds the two files:

target_dbnode=host-vak332
scp /home/opc/oracle.mgmt_agent.260729.1846.Linux-x86_64.zip \
  /home/opc/mgmt_agent_exadbd.rsp \
  opc@"$target_dbnode":/home/opc/
ssh opc@"$target_dbnode" \
  'chmod 0600 /home/opc/mgmt_agent_exadbd.rsp'

Treat the copied response file as a template. On the target node, change AgentDisplayName to a node-specific value before staging the file in /tmp. This makes every database node easy to identify in the Management Agent and Log Analytics consoles. For example:

AgentDisplayName = odb-aws-host-vak332

Retain the validated install key and Service.plugin.logan.download=true, along with any approved tags or gateway settings that apply to that node. Then repeat Steps 2–5 on the target node: complete the preflight checks, create the /u02 target and /opt/oracle/mgmt_agent symlink, extract the ZIP, stage the response file, run installer.sh, validate packageType=ZIP, and confirm that the agent and logan plug-in are active. Apply the emd.properties and 2 GB heap baseline below, restart the agent, and validate log collection from that node.

Repeat this sequence for every database node in every VM cluster. Confirm a distinct active agent and current log records for each node in OCI Log Analytics. After a node is registered successfully, delete its temporary response-file copies as described in Step 4. Keep any retained master copy only in the approved secret-management location.

Best practice: establish an Exadata Management Agent baseline

For an ExaDB-D or ExaC@C fleet that collects database and GI logs, start with the following Management Agent properties. Back up emd.properties, edit the original file, and add the settings in its additional-properties section:

sudo cp -p \
  /opt/oracle/mgmt_agent/agent_inst/config/emd.properties \
  /opt/oracle/mgmt_agent/agent_inst/config/emd.properties.pre-exadbd-baseline
sudo vi /opt/oracle/mgmt_agent/agent_inst/config/emd.properties

loganalytics.enable_large_dir=true

# Sender properties
senderManagerMaxDiskUsedTotal=1000
_senderManagerPoolSize=10

# Connection pooling properties
clientConnectionPool.SERVICE_INVOKING.defaultMaxConnectionsPerRoute=50
clientConnectionPool.SERVICE_INVOKING.maxTotalConnections=100

Treat these values as the starting baseline for this Exadata use case. Adjust them according to the number of VM clusters, databases, PDBs, enabled log sources, files created per hour, and observed collection backlog. Validate changes first on a representative VM cluster and monitor agent CPU, memory, disk use, collection latency, and errors before applying them across the fleet. The larger examples in the earlier A-Team sizing table are planning references rather than a universal certified profile.

Oracle defines a large log directory as one containing more than 10,000 files. loganalytics.enable_large_dir=true allows the agent to monitor such directories, but it can increase heap, disk, inode, and open-file demand. Clean up files that no longer need collection and confirm host capacity before enabling high-volume sources. Large-directory configuration

For this standalone Management Agent deployment, configure a 2 GB maximum Java heap as the Exadata fleet baseline. Oracle documents 512 MB as the out-of-box maximum heap for a standalone Management Agent; 2 GB is the operational baseline used in this article, not the product default. Back up java.options, confirm the current active -Xmx entry, and replace it with -Xmx2048m:

sudo cp -p \
  /opt/oracle/mgmt_agent/agent_inst/config/java.options \
  /opt/oracle/mgmt_agent/agent_inst/config/java.options.pre-2gb-heap
sudo grep -n '^-Xmx' \
  /opt/oracle/mgmt_agent/agent_inst/config/java.options
sudo vi /opt/oracle/mgmt_agent/agent_inst/config/java.options

-Xmx2048m

Restart the agent once after both files are updated, then verify the service, heap entry, and Management Agent properties:

sudo systemctl restart mgmt_agent
sudo systemctl status mgmt_agent --no-pager
sudo grep -n '^-Xmx' \
  /opt/oracle/mgmt_agent/agent_inst/config/java.options
sudo grep -E \
  '^(loganalytics\.enable_large_dir|senderManagerMaxDiskUsedTotal|_senderManagerPoolSize|clientConnectionPool\.SERVICE_INVOKING\.(defaultMaxConnectionsPerRoute|maxTotalConnections))=' \
  /opt/oracle/mgmt_agent/agent_inst/config/emd.properties

The following output from host-vak331 confirms the physical /u02 configuration path and the configured 2 GB heap. The -XX:+HeapDumpOnOutOfMemoryError option was also present in the installed file:

[root@host-vak331 config]# pwd
/u02/app/oracle/mgmt_agent/agent_inst/config

[root@host-vak331 config]# cat java.options
#encoding=UTF-8

#These java.options contain user supplied extensions to the options set in wrapper.conf.

#If you modify this file, please make sure that you restart the mgmt_agent service.

#Note: Please do not delete this file. Deleting this file will cause agent failures.

#Add each additional parameter on new line

#------------------------------------------------------------------------------------------------
-Dpolyglot.engine.AllowExperimentalOptions=true
-Dpolyglot.js.nashorn-compat=true
-Djdk.tls.client.protocols=TLSv1.3,TLSv1.2
-Djdk.tls.server.protocols=TLSv1.3,TLSv1.2

-Doracle.polaris.fips.approved_only=false
-XX:+HeapDumpOnOutOfMemoryError
-Dio.netty.native.workdir=/u02/app/oracle/mgmt_agent/agent_inst/tmpdir
-Dio.netty.tmpdir=/u02/app/oracle/mgmt_agent/agent_inst/tmpdir

-Dorg.newsclub.net.unix.library.tmpdir=/u02/app/oracle/mgmt_agent/agent_inst/tmpdir

-Dorg.bouncycastle.native.cpu_variant=java
-Xmx2048m

The same host showed the requested Log Analytics sender and connection-pool baseline at the end of emd.properties:

[root@host-vak331 config]# tail -10 emd.properties
################################
loganalytics.enable_large_dir=true

#Sender properties
senderManagerMaxDiskUsedTotal=1000
_senderManagerPoolSize=10

# Connection pooling properties

clientConnectionPool.SERVICE_INVOKING.defaultMaxConnectionsPerRoute=50
clientConnectionPool.SERVICE_INVOKING.maxTotalConnections=100
[root@host-vak331 config]#

The file output above validates the saved configuration. After the restart, the following systemctl output confirmed that the service was enabled and active. It also showed the wrapper and Java process running from the physical /u02/app/oracle/mgmt_agent installation. Long process arguments were truncated by systemctl in the captured terminal:

[root@host-vak331 config]# sudo systemctl status mgmt_agent --no-pager
● mgmt_agent.service - mgmt_agent
   Loaded: loaded (/etc/systemd/system/mgmt_agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2026-09-17 20:08:40 UTC; 3s ago
  Process: 348607 ExecStop=/u02/app/oracle/mgmt_agent/agent_inst/bin/agentcore stop sysd (code=exited, status=0/SUCCESS)
  Process: 350167 ExecStart=/u02/app/oracle/mgmt_agent/agent_inst/bin/agentcore start sysd (code=exited, status=0/SUCCESS)
 Main PID: 350257 (wrapper)
    Tasks: 73 (limit: 319999)
   Memory: 562.9M
   CGroup: /system.slice/mgmt_agent.service
           ├─350257 /u02/app/oracle/mgmt_agent/agent_inst/bin/./wrapper /u02/app/oracle/mgmt_agent/agent_inst/bin/../config/wrapper.conf wrapper.syslog.ident=mgmt_agent wrapper.pidfile=/u02/app/oracle/mgm…
           └─350326 /usr/lib/jvm/jdk-1.8.0_491-oracle-x64/bin/java -Dorg.tanukisoftware.wrapper.WrapperSimpleApp.maxStartMainWait=5 -Djava.security.egd=file:///dev/./urandom -Dpolyglot.engine.AllowExperim…

Sep 17 20:08:28 host-vak331 systemd[1]: Starting mgmt_agent...
Sep 17 20:08:28 host-vak331 agentcore[350167]: Starting mgmt_agent...
Sep 17 20:08:35 host-vak331 agentcore[350167]: Waiting for mgmt_agent.........
Sep 17 20:08:40 host-vak331 agentcore[350167]: .....running: PID:350257
Sep 17 20:08:40 host-vak331 systemd[1]: Started mgmt_agent.

Together, the saved-file output and this post-restart service status validate the 2 GB heap and Management Agent property change on host-vak331. Continue monitoring memory, agent logs, and collection backlog as workload is added.

Troubleshoot: the agent reports OutOfMemoryException

An agent can run out of heap when its assigned collection load exceeds the configured maximum. Check the active -Xmx value and recent service messages first:

sudo grep -n '^-Xmx' \
  /opt/oracle/mgmt_agent/agent_inst/config/java.options
sudo journalctl -u mgmt_agent -n 100 --no-pager

If the standalone agent is still using the 512 MB default, apply the 2 GB procedure above, restart the service, and verify that it remains active. If the exception continues at 2 GB, confirm that the VM has sufficient available memory, then review file counts, log-generation rate, enabled sources, collection backlog, and other plug-in workloads before increasing the heap again. Preserve the agent logs and engage Oracle Support when the failure persists with a supported configuration. Management Agent OutOfMemoryException troubleshooting

After successful registration, remove the response file and downloaded installation artifacts from their transfer locations, or store them in the approved secret location. Record node, agent OCID, source associations, and verification time before repeating the process. For ExaC@C, retain the ZIP and /u02 approach; adapt connectivity to the customer’s approved OCI egress path or Management Gateway, then apply the same collection checks.

Adapt the deployment prerequisites for OCI ExaDB-D and ExaC@C

The source/entity configuration and verification steps are reusable across the fleet. Apply each environment’s network and support requirements before reusing the commands:

Use the same per-VM agent, log-access, source-association, and recent-record checks in each deployment. Follow the current installation guidance and host/network prerequisites.

Associate database and GI logs with the right entities

After the agent and plug-in are available, build the collection model for each VM. Use the actual source definitions in your tenancy, including their required entity properties and file patterns. The following Oracle-defined sources provide a useful starting point. Source catalog

For database diagnostics, connect through your approved DBA session and discover the active ADR paths:

SELECT name, value
FROM v$diag_info
ORDER BY name;

Record the ADR home and trace directory for the particular instance. Repeat discovery for other instances instead of copying the first node’s path. Listener diagnostics should be mapped to the appropriate local or SCAN listener entity. Clusterware’s documented ADR home is <GRID_ORACLE_BASE>/diag/crs/<hostname>/crs, with the text alert in trace/alert.log; discover the actual Grid base and hostname on the VM. Original database walkthrough, Clusterware diagnostics

Grant mgmt_agent access to database and listener logs

The standalone Management Agent runs as mgmt_agent. That account needs read access to the log files and read/execute access through every directory in each log path. Oracle recommends filesystem ACLs as the first method to grant this access. Review existing ACLs before changing them, verify that the acl package is present, and add permissions without replacing existing entries. The command is setfacl; the similar spelling setfactl is invalid. Grant read access to the agent user

The following database-alert example uses the observed /u02/app/oracle/diag/rdbms/oradbrac/ORADBRAC2/trace directory. Replace the database unique name, instance name, and path with the values discovered on the target database node. Open a root login shell:

sudo su -

At the root prompt, grant traversal on the parent directories, grant recursive access to the existing ADR contents, and set a default ACL on the trace directory for future files:

rpm -q acl
getfacl -p /u02/app/oracle/diag
setfacl -m u:mgmt_agent:rx /u02
setfacl -m u:mgmt_agent:rx /u02/app
setfacl -m u:mgmt_agent:rx /u02/app/oracle
setfacl -m u:mgmt_agent:rx /u02/app/oracle/diag
setfacl -R -m u:mgmt_agent:rx /u02/app/oracle/diag
setfacl -d -m u:mgmt_agent:rx \
  /u02/app/oracle/diag/rdbms/oradbrac/ORADBRAC2/trace
getfacl -p \
  /u02/app/oracle/diag/rdbms/oradbrac/ORADBRAC2/trace
exit

The following listener example is for host-vak332. Replace the hostname and retain only the listener directories that exist on the target node:

sudo su -

At the root prompt, run:

rpm -q acl
getfacl -p /u01/app/grid/diag
setfacl -m u:mgmt_agent:rx /u01
setfacl -m u:mgmt_agent:rx /u01/app
setfacl -m u:mgmt_agent:rx /u01/app/grid
setfacl -m u:mgmt_agent:rx /u01/app/grid/diag
setfacl -R -m u:mgmt_agent:rx /u01/app/grid/diag
setfacl -d -m u:mgmt_agent:rx \
  /u01/app/grid/diag/tnslsnr/host-vak332/listener/trace
setfacl -d -m u:mgmt_agent:rx \
  /u01/app/grid/diag/tnslsnr/host-vak332/listener_scan1/trace
setfacl -d -m u:mgmt_agent:rx \
  /u01/app/grid/diag/tnslsnr/host-vak332/listener_scan2/trace
setfacl -d -m u:mgmt_agent:rx \
  /u01/app/grid/diag/tnslsnr/host-vak332/listener_scan3/trace
setfacl -d -m u:mgmt_agent:rx \
  /u01/app/grid/diag/tnslsnr/host-vak332/asmnet1lsnr_asm/trace
getfacl -p \
  /u01/app/grid/diag/tnslsnr/host-vak332/listener/trace
exit

Validate the effective permissions as the agent account before associating a source. The loop below checks read and traversal access to the example directories without displaying log contents:

for log_dir in \
  /u02/app/oracle/diag/rdbms/oradbrac/ORADBRAC2/trace \
  /u01/app/grid/diag/tnslsnr/host-vak332/listener/trace \
  /u01/app/grid/diag/tnslsnr/host-vak332/listener_scan1/trace \
  /u01/app/grid/diag/tnslsnr/host-vak332/listener_scan2/trace \
  /u01/app/grid/diag/tnslsnr/host-vak332/listener_scan3/trace \
  /u01/app/grid/diag/tnslsnr/host-vak332/asmnet1lsnr_asm/trace
do
  sudo -u mgmt_agent test -r "$log_dir"
  sudo -u mgmt_agent test -x "$log_dir"
done

After confirming the actual filenames, read one line from a representative database alert log and listener log as mgmt_agent. Successful output proves file-level read access without relying on root privileges:

database_alert_log=/u02/app/oracle/diag/rdbms/oradbrac/ORADBRAC2/trace/alert_ORADBRAC2.log
listener_log=/u01/app/grid/diag/tnslsnr/host-vak332/listener/trace/listener.log
sudo -u mgmt_agent tail -n 1 "$database_alert_log"
sudo -u mgmt_agent tail -n 1 "$listener_log"

Repeat the ACL and validation steps on every database node using that node’s discovered paths. Recheck access after database or Grid homes change and after log rotation. The default ACL helps future files inherit access, while the recursive ACL covers files and directories that already exist.

Create the ORADBRAC1 Oracle Database Instance entity

Create a separate entity for each database instance so that the Oracle-defined alert-log source can resolve the correct paths and use the Management Agent on the node that owns those files. This example creates ORADBRAC1 on host-vak331; the earlier ORADBRAC2 ACL path is a separate node-specific example.

1. Open Observability & Management → Log Analytics → Administration → Entities, select the entity compartment, and click Create.
2. Enter ORADBRAC1 for Name and select Oracle Database Instance for Entity Type.
3. Select the Management Agent compartment and odb-aws-host-vak331 as the Management Agent.
4. Enter the values discovered on that database node. The captured configuration uses the following properties:

1. Click Save Changes, reopen the entity, and confirm that its type, agent, and properties match the target instance. Oracle-defined file sources use entity properties in their include patterns, so missing or incorrect property values can prevent source association or file matching. Create the database entity, entity properties in source patterns

Figure 7. Saved ORADBRAC1 entity used for Oracle Database alert-log collection on host-vak331. The entity type is Oracle Database Instance, the associated agent is odb-aws-host-vak331, and the node-specific diagnostic properties are populated. The compartment identifier and entity OCID were replaced for publication.

Create the ORADBRAC2 Oracle Database Instance entity

Repeat the entity creation for the second database instance, using the Management Agent and Oracle home on host-vak332:

Review the values against the paths discovered on host-vak332, then click Create Entity. After creation, reopen ORADBRAC2 and confirm that the selected agent and properties were saved before associating a source.

Figure 8. Creation of the ORADBRAC2 entity for alert-log collection on host-vak332. The entity uses odb-aws-host-vak332, the node’s /u02 diagnostic paths, dbhome_1, and port 1521. The compartment hierarchy and private DNS suffixes were replaced for publication.

Associate the database instances and log sources with Add Data

Use the Add Data wizard to create the source-entity associations for both RAC instances in one workflow. Oracle documents this wizard as the continuous-collection path for selecting entities, selecting sources, assigning a log group, and validating the association. Set up continuous log collection

1. Open Observability & Management → Log Analytics → Administration → Overview, select the compartment that contains the database entities, and click Add Data.

Figure 9. Start the source-association workflow from Add Data. The compartment name and hierarchy were replaced for publication.

1. Expand Monitor with management agent and select Custom Selection.

Figure 10. Custom Selection opens the Configure Agent-based Log Collection workflow for explicitly selected entities and sources.

1. On Select Entities, filter to Oracle Database Instance when the fleet contains many entity types. Select ORADBRAC1 and ORADBRAC2, confirm that each row resolves to its node-local Management Agent, and click Next.

Figure 11. Both RAC instance entities are selected. Private DNS suffixes and Management Agent OCID fragments were replaced with example hostnames and the corresponding agent display names.

1. On Select Sources, select the Oracle-defined Database Alert Logs file source. That source is the baseline for this walkthrough. The captured environment also selected Database Attention Logs and Database Incident Dump Files; add those only when the operating requirement and expected ingestion volume justify them.

Figure 12. Database Alert Logs is the required source for this walkthrough. Database Attention Logs and Database Incident Dump Files are optional selections visible in the captured configuration.

1. In Log Group Compartment, choose the intended destination compartment. Select an existing log group, or click Create New. This example creates Multicloud_OracleDB_LogGroup with the description OracleDB@AWS.

Figure 13. Creation of the Multicloud_OracleDB_LogGroup destination. The compartment identifier and hierarchy were replaced for publication.

1. Confirm that the new log group is selected, review the entity and source choices, and click Validate and configure log collection.

Figure 14. Pre-submit configuration with the destination log group selected. This screen does not prove that validation or association creation completed.

1. Wait for the wizard to display its success confirmation. Open the activity status or source name and verify that the associations for both entities completed successfully. Then use Take me to Log Explorer or open Log Explorer separately. The supplied screenshot set stops at the pre-submit screen in Figure 14, so retain the actual confirmation or work-request result as deployment evidence before publication or rollout acceptance.

If validation fails, do not repeatedly submit the same configuration. Inspect Agent Collection Warnings, verify that the entity properties resolve to the intended alert-log paths, test file access as mgmt_agent, confirm that the Log Analytics plug-in is running, and verify the dynamic group and IAM policy below.

Verify the Management Agent dynamic group and upload policy

The Set Up Ingestion workflow can create some IAM resources automatically, but verify the result rather than assuming that it selected the intended scope. In Identity & Security → Dynamic Groups, open exadb-mgmtagent-dg and confirm that its matching rule covers the Management Agent resources in the Management Agent compartment:

ALL {resource.type='managementagent', resource.compartment.id='<management-agent-compartment-OCID>'}

Avoid the broader ALL {resource.type='managementagent'} rule unless every Management Agent in the tenancy is intentionally part of the collection identity. If the dynamic group belongs to an identity domain, include the domain name in policy subjects as Oracle documents. Continuous collection IAM

Oracle’s Allow continuous log collection using management agent dynamic groups template contains the metrics and upload permissions. This article’s policy collection also includes tenancy-level READ USERS, which appears in Oracle’s automatic agent-ingestion policy example. Select the compartment that contains the target Log Analytics resources for the compartment-scoped grants:

ALLOW DYNAMIC-GROUP <identity-domain>/exadb-mgmtagent-dg TO USE METRICS IN COMPARTMENT <log-analytics-compartment>
ALLOW DYNAMIC-GROUP <identity-domain>/exadb-mgmtagent-dg TO {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} IN COMPARTMENT <log-analytics-compartment>
ALLOW DYNAMIC-GROUP <identity-domain>/exadb-mgmtagent-dg TO READ USERS IN TENANCY

Omit <identity-domain>/ when the subject syntax for the tenancy does not require it. The captured environment uses the valid but broader IN TENANCY form shown in Figure 15. Prefer the narrowest compartment scope that covers the metrics and upload operations; READ USERS remains tenancy-scoped. Oracle-defined policy templates, automatic-ingestion policy example

Figure 15. The captured tenancy-wide policy grants exadb-mgmtagent-dg permission to emit metrics and upload logs to Log Analytics log groups. The additional tenancy-scoped READ USERS statement is included in the policy collection shown in the text above. The tenancy OCID, cross-tenancy statements, and unrelated policy lines were removed from the publication copy.

Complete the association check in Log Analytics:

1. Reopen ORADBRAC1 and ORADBRAC2 under Administration → Entities and confirm that Database Alert Logs appears under Associated Sources.
2. Inspect the source’s resolved include patterns and exclusions. Confirm that they resolve to the ORADBRAC1 alert-log location on host-vak331 and the ORADBRAC2 alert-log location on host-vak332.
3. Inspect Agent Collection Warnings and correct entity-property, file-matching, IAM, or mgmt_agent access problems before proceeding.
4. Open Log Explorer, select Multicloud_OracleDB_LogGroup and Database Alert Logs, and verify a recent record from both ORADBRAC1 and ORADBRAC2, including each record’s timestamp, message, and extracted fields.

Figure 16. Entity-scoped collection evidence for ORADBRAC1. Log Explorer shows 428 Database Alert Logs records in the selected 14-day window, including parsed entity, entity type, source, priority, and label context. This validates records for this entity and time range; repeat the check for ORADBRAC2 and after log rotation.

The file collection workflow explains source patterns and entity associations. Begin with one alert representation; collecting both text and XML copies can duplicate an event in downstream counts. Extend to traces or incident dumps when their diagnostic value justifies the volume.

For rollout acceptance, capture the expected VM/entity/source list alongside agent status, collection warnings, and a recent record from each enabled source. A source with no recent records may simply be quiet. Compare collection status and file activity before concluding that it is either healthy or broken.

Optional extension: ExaWatcher diagnostics

ExaWatcher can add guest operating-system context to a database investigation, but its collection and parsing workflow is outside this article’s database and GI focus. For a worked implementation, see Jerry Russell’s Creating an Advanced Log Parser in Oracle Log Analytics for Exawatcher VMStat Output. It demonstrates parser functions, source data filters, Management Agent association, on-demand testing, and visualizations for ExaWatcher VMStat records.

Before applying that pattern, validate the ExaWatcher file location, agent permissions, timestamp handling, rotation behavior, and VMStat semantics on the target guest image. Oracle’s ExaWatcher documentation provides the utility background and operational guidance.

Build fleet dashboards and investigation queries

Start with the Oracle Database Alert dashboard, which includes views of database alert trends and errors. Duplicate the Oracle-defined dashboard to customize it, then add your guest-host and GI views. Configure common time and scope filters so the panels describe the same incident window. Database Alert dashboard, Dashboard customization

Open the Oracle Database Alert dashboard

Open Log Analytics → Dashboards, select the intended compartment, and search for oracle database. The Created by column identifies the Oracle-defined dashboards. Open Oracle Database Alert.

Figure 17. Oracle-defined database dashboards available in Log Analytics. The compartment name was replaced for publication.

Oracle-defined dashboards cannot be edited directly. Duplicate Oracle Database Alert before adding fleet-specific widgets, changing its filters, or replacing its saved searches. Apply the log-group compartment, region, entity compartment, database entities, and time range that belong to the ExaDB-D fleet.

Build the Oracle Database Alert Error Summary saved search

Open the Error Summary widget in Log Explorer or create a saved search using this query:

'Log Source' = 'Database Alert Logs'
| link 'Error ID', Entity
| lookup table = 'ORA Error Messages' select errmsg as 'Error Message'
  using 'Error ID' = errid
| stats unique('Problem Priority') as Severity
| eventstats ifnull(sum(Count), 0) as 'Total Errors',
  unique(Entity) as 'Database(s) with issues',
  ifnull(distinctcount(Entity), 0) as 'Number of database(s) with issues'
| fields target = ui -'Total Errors', -'Number of database(s) with issues'

The link command groups records by error ID and database entity. The lookup enriches matching error IDs with the ORA Error Messages table, while eventstats adds totals across the result. The final fields target = ui clause removes the two helper totals from the Link groups table without discarding the backend fields. If Error Message is blank, confirm that the lookup contains the parsed errid; keep the parsed Error ID and original records as the primary evidence.

Figure 18. The saved search groups two ORA-7445 records for oradbrac1, marks the group High severity, and reports one database with issues in the selected 14-day window.

Save the search with a relative time range when the widget should stay current, and add it to the duplicated dashboard. A fixed time range preserves one investigation window instead. Oracle documents saved searches as reusable dashboard widgets and supports duplicating Oracle-defined searches before customization. Save and share log searches

Drill into the ORA-7445 records

In Figure 18, click the linked count 2 for the ORA-7445 row. Log Analytics opens the two records in the Link collection and narrows the time range to their interval.

Figure 19. Record-level drilldown from the ORA-7445 group. The Link collection identifier and patch inventory were replaced for publication; the entity, source, severity, label, timestamps, and two-record sequence remain visible.

Use this drilldown to inspect the alert entry, trace-file or incident references, surrounding database messages, and records from GI, ASM, and listener sources in the same window. The two linked records establish temporal context; they do not by themselves identify the ORA-7445 root cause. Preserve the original alert and trace evidence for the database diagnostic or Oracle Support workflow.

Add AI-powered assistance with LoganAI

LoganAI is the generative AI capability in Log Analytics. It can summarize individual records, groups, clusters, and charts; explain complex results; and propose context-aware follow-up questions within the Log Analytics workflow. It uses OCI Generative AI in the tenancy. Oracle cautions that AI-generated responses can be incomplete or inaccurate, so validate every explanation against the original records before making an operational decision. Use LoganAI

To enable it for a region, open Administration → Service Details → Configure LoganAI settings. Select the OCI Generative AI region and compartment, choose the model behavior, review and accept the required model licenses, select the user license-acceptance policy, and configure the displayed IAM permissions. Run Test under Validate settings before clicking Update LoganAI settings. LoganAI availability is regional, the selected OCI Generative AI service can add usage charges, and logs can be sent from the Log Analytics region to the selected Generative AI region. Enable LoganAI

Figure 20. Pre-update LoganAI configuration. The service namespace was replaced for publication. The background status remains Disabled; this screen documents the intended settings and does not prove that validation or activation succeeded.

After activation, use Explain on the selected ORA-7445 group or its surrounding chart to request a concise summary and investigation questions. Check every claim against the records, database trace files, and corroborating sources before adding it to the incident timeline or RCA.

Use a consolidated database-alert view

The Oracle-defined dashboard brings the alert summary, new issues and outliers, log trends, ORA and TNS distributions, corrupt-block results, database distribution, severity, and error trends into one view. Set the time and scope filters before interpreting its totals.

Figure 21. Oracle Database Alert dashboard populated in a sample environment. Use its widget layout as the starting point, then scope the duplicated dashboard to the ExaDB-D log group and database entities. The displayed sample counts and entity names are not results from the ORADBRAC1 and ORADBRAC2 walkthrough.

For the multicloud operations view, extend the duplicate with collection coverage, listener failures, and GI and ASM events. Keep the same time, log-group, and entity scope across widgets so operators can move from a fleet-level symptom to the contributing database and infrastructure records.

Turn the dashboard into an operating routine

For the connection-failure scenario, use the database and listener panels to identify affected instances, then inspect GI and ASM records in the same window. Ask LoganAI to summarize the selected evidence and suggest the next investigation step; verify its response against the original records before recording an RCA conclusion.

Save useful searches as dashboard widgets, validate that their filters apply to every intended source, and document what each count means. Once the query and threshold have been tested, selected searches can feed scheduled detection and alerting workflows. Scheduled searches

Track the collection pipeline as part of the same routine. Review agent health, warnings, new VM onboarding, and log volume. Retain frequently used diagnostics for the required investigation period, and add high-volume traces with a clear use case. These decisions keep the fleet dashboard useful as the environment changes.

Conclusion

OCI Log Analytics gives Oracle Database@AWS ExaDB-D and ExaC@C operations a common way to work with distributed diagnostic evidence. Its value comes from connecting Oracle-aware source configuration with repeatable investigation: identify an affected instance, follow the related database and GI messages, inspect guest-host evidence, and share a query that another operator can reproduce.

Begin with one VM cluster. Validate the ZIP installation, environment-specific network path, source associations, file rotation, timestamps, and extracted fields, then expand the validated database and GI collection pattern across the Oracle Database@AWS or ExaC@C fleet.

References

• Maximizing Visibility: Collecting and Analyzing Logs from Oracle Databases using OCI Logging Analytics
• Creating an Advanced Log Parser in Oracle Log Analytics for Exawatcher VMStat Output

In the legacy operations model, a pager sounds, a human engineer joins a bridge, and the first critical minutes are spent collecting context. Which system changed? Which database slowed down? Which application tier is producing errors? Which metric crossed a threshold? Which log message matters? Which business process is impacted? That model worked when environments were smaller, changes were slower, and operational knowledge could fit inside a few expert teams. It becomes fragile in a cloud operating model where applications are distributed, infrastructure is elastic, data platforms are mission critical, and signals arrive from many layers at the same time.

DevOps changed the industry by connecting development and operations: if you build it, you run it. But AI transformation is changing the operating model again. AI agents can now assist with code generation, application development, testing, and release engineering. The next logical step is AI Ops: using AI to operate the application, the infrastructure, the database estate, and the cloud platform itself.

The goal is not to remove people from operations. The goal is to move human experts out of repetitive correlation work and into architecture, governance, exception handling, and business decision-making. Autonomous resilience is the target state: systems that can sense, diagnose, plan, remediate, verify, and learn with appropriate human supervision.

What AI Ops means

AI Ops is the application of AI, automation, observability, and operational governance to run modern digital systems more intelligently. It is not just a chatbot attached to monitoring data. It is not a folder of scripts triggered by alerts. It is an architecture where telemetry, policies, agents, evaluators, workflows, and incident processes work together.

In a mature AI Ops model, the system can:

• Detect a performance, availability, capacity, security, or data-platform anomaly.
• Correlate signals across metrics, logs, events, traces, SQL data, and configuration context.
• Determine whether the event is noise, a real incident, a capacity risk, or an emerging business-impacting condition.
• Generate a remediation plan that explains both the technical action and the expected outcome.
• Validate the plan before execution through an evaluator or policy guardrail.
• Execute the approved remediation through controlled OCI services and APIs.
• Verify the post-execution result against the desired state.
• Produce an incident summary, problem record, and root cause analysis for future learning.

The key shift is that operations becomes an agentic loop, not a manual search exercise.

Figure 1: OCI AI Ops framework for autonomous resilience, from observability data foundation to agentic remediation and problem management.

Observability is the data foundation

AI Ops begins with observability. An agent cannot diagnose what it cannot perceive. A framework that only sees CPU utilization cannot understand user experience. A framework that only sees logs cannot understand capacity. A framework that ignores database state cannot make reliable decisions for enterprise workloads.

The data foundation must include several classes of operational evidence:

• Metrics: numerical time-series data such as CPU, memory, storage, latency, queue depth, throughput, and error rate.
• Logs: detailed records from applications, infrastructure, databases, middleware, security systems, and integration platforms.
• Events: discrete changes in cloud resources, lifecycle state, configuration, alarms, and operational activity.
• Traces: end-to-end transaction paths across applications, APIs, services, and dependencies.
• SQL data: database-level state, workload, waits, sessions, execution patterns, backup posture, capacity, and performance signals.
• Topology and resource context: relationships across applications, compute, databases, networks, storage, tags, compartments, and business services.

For enterprise customers, this foundation is not only about troubleshooting. It is how the business understands operational risk, service performance, cost exposure, capacity pressure, and resilience posture.

OCI Observability and Management as the AI Ops service fabric

OCI provides a strong set of Observability and Management services that can become the service fabric for an AI Ops framework. Each service contributes a different kind of operational signal or control point.

When these services are connected into an architecture, they do more than generate dashboards. They create an operational decision fabric. Customers can use that fabric to understand which systems are at risk, which capacity trends require investment, which incidents have the highest business impact, and which remediations can be safely automated.

The database-native AI and context plane

Observability tells the agent what is happening. Autonomous operations also require a governed context plane that can explain what the signals mean, how resources are related, which prior knowledge applies, and what actions are available. This is where Oracle AI Database and its AI and machine-learning capabilities extend the OCI Observability and Management service fabric.

The design should not force every source artifact into one storage system. Large log archives, documents, images, video, runbooks, and incident attachments can remain in OCI Object Storage or an enterprise data lake. Oracle Autonomous AI Lakehouse can query data in object storage through external and hybrid-partitioned tables. Oracle AI Database can then hold the structured operational facts, document chunks, embeddings, source references, resource identities, and relationships required for governed retrieval and analysis.

These are not separate AI islands. Together they form a converged context and agent platform:

Figure 2: The Oracle AI Database context plane grounds Private Agent Factory workflows in operational evidence, enterprise knowledge, resource relationships, and governed retrieval.

The context plane grounds the workflow in governed operational evidence. Oracle AI Database Private Agent Factory operationalizes that context by providing the environment to build, test, govern, deploy, and operate Select AI Agent and approved analytical workflows.

How Private Agent Factory and the Oracle AI services fit together

The components in this architecture have distinct responsibilities:

• OCI Object Storage and Oracle Autonomous AI Lakehouse retain large source artifacts and historical operational data without forcing every document, log archive, image, or video into the database.
• Oracle AI Database and AI Vector Search provide the governed operational context: structured facts, JSON, embeddings, resource identities, graph relationships, source references, and hybrid retrieval.
• OCI Generative AI provides OCI-managed foundation-model inference and enterprise agent services, including chat, embeddings, rerank, OpenAI-compatible APIs, the OCI Responses API, agent tools, vector stores, memory, and hosted agentic applications. In this reference architecture, it is a candidate approved model provider or a separate OCI-managed agent runtime; the target integration must be validated and should cross explicit API or MCP boundaries.
• Oracle Private AI Services Container provides private embedding, reranking, classification, and LLM inference services in a customer-controlled environment. Its Vector Index Service can use GPU infrastructure to accelerate index creation without turning the container into the system of record.
• Oracle AI Database Private Agent Factory is the agent engineering and lifecycle-control environment. It brings together enterprise content, approved model providers, APIs, MCP tools, Select AI Agent, and analytical workflows for building, testing, governing, deploying, and operating agents.
• Select AI Agent is the database-native agent runtime and orchestration capability. DBMS_CLOUD_AI_AGENT manages tools, tasks, agents, teams, and multi-step workflows that can use SQL, RAG, web search, and notification tools.
• Autonomous AI Database MCP Server exposes approved Select AI Agent tools through a managed, per-database MCP endpoint while retaining database authentication, authorization, auditing, and least-privilege controls.
• Oracle Machine Learning Data Science Agent complements the agent runtime with human-supervised data profiling, outlier analysis, statistical investigation, model training, evaluation, and inference.

Private Agent Factory does not replace the context plane, the evaluator, or the controlled execution boundary. It operationalizes the agents that use the context. The evaluator remains an independent decision gate that checks evidence, policy, risk, confidence, approval requirements, and blast radius before any operational action is allowed to proceed.

The source evidence remains authoritative. Embeddings are derived retrieval structures, graph edges are relationship context, and model-generated conclusions remain hypotheses until the evaluator verifies them against fresh telemetry.

NVIDIA AI Blueprints as reference accelerators

NVIDIA AI Blueprints provide useful implementation patterns for parts of this architecture. They should be treated as adaptable reference solutions rather than packaged integrations with Select AI Agent.

• The NVIDIA RAG Blueprint demonstrates multimodal extraction, chunking, embeddings, hybrid retrieval, reranking, guardrails, evaluation, and agentic RAG. In an Oracle-centered implementation, these patterns can feed evidence and metadata into Oracle AI Vector Search, while Select AI Agent uses approved SQL and RAG tools to interact with the governed database context.
• The NVIDIA AI-Q Blueprint demonstrates bounded research, planning, parallel investigation, source selection, citation-backed synthesis, and evaluation. Those patterns are useful for an AI Ops investigator that must assemble evidence before proposing remediation. Oracle database tools can be exposed to such workflows through governed MCP or API boundaries.
• The NVIDIA Video Search and Summarization Blueprint demonstrates how video can become searchable operational evidence through vision models, embeddings, metadata streams, alert verification, and MCP-accessible tools. This is relevant to data centers, manufacturing, transportation, physical security, and other environments where an outage or anomaly may have visual evidence.

The NVIDIA component name is cuVS, not an embedding model. Oracle Private AI Services Container provides embedding services; its Vector Index Service uses NVIDIA GPUs and NVIDIA cuVS to accelerate vector-index construction. Keeping this distinction clear separates model inference, embedding generation, vector storage, index construction, and retrieval into explicit architectural responsibilities.

From dashboards to decisions

Traditional observability often stops at visibility. A dashboard shows that CPU is high, a database wait event is increasing, a transaction is slower than normal, or a log pattern is repeating. That visibility is useful, but it still leaves the human operator responsible for correlation, prioritization, and action.

AI Ops should move customers from dashboards to decisions.

For example:

• OCI Monitoring identifies a compute CPU saturation pattern.
• APM shows that the affected compute instance is part of a customer-facing application path.
• Log Analytics detects a repeating application error after a deployment.
• Database Management shows increased database sessions and wait events.
• Ops Insights indicates the workload has been trending toward a capacity ceiling for several weeks.
• Resource Analytics identifies the affected resource, its OCI configuration, and its known relationships to dependent infrastructure.
• A metrics overlay binds CPU, memory, latency, error rate, and change events to the corresponding graph vertices and time window.
• A graph traversal follows the dependency path to estimate which hosts, databases, application services, and business processes may be inside the blast radius.
• Oracle AI Vector Search retrieves relevant runbooks, previous incidents, change records, and technical knowledge, while Select AI Agent uses approved SQL and RAG tools to investigate the structured and unstructured operational context.

Individually, these are signals. Together, they tell a business story: which service is affected, why it matters, whether this is an incident or a capacity trend, what remediation is available, and which investment decision should be made next.

That is where AI Ops creates business value. It compresses the time between signal, understanding, action, and learning.

OCI reference architecture for autonomous resilience

The OCI AI Ops framework can be organized into five architectural layers.

Figure 3: OCI observability signals move through event transport and processing services before invoking the governed agent workflow.

1. Telemetry and context collection

The collector layer gathers operational evidence from OCI Monitoring, Logging, Log Analytics, Application Performance Monitoring, Database Management, Ops Insights, Resource Analytics, Events, traces, SQL diagnostics, and resource metadata. It can also register unstructured evidence from Object Storage or an enterprise data lake and create embeddings for approved content through Oracle Private AI Services Container.

This layer should normalize identity and context as much as possible. Resource OCIDs, compartment names, tags, application names, database names, service tiers, environment labels, and business ownership all matter. Each chunk, embedding, metric, event, and graph vertex should retain a reference to its source and resource identity. Without this context, an agent may understand the symptom but miss the business priority or retrieve evidence for the wrong environment.

2. Detection and triggering

The detector layer turns telemetry into operational signals. OCI Monitoring alarms, OCI Events rules, Log Analytics patterns, APM thresholds, database performance indicators, capacity forecasts, and security anomalies can all become triggers.

In a traditional model, the trigger may send an email or open a ticket. In an AI Ops model, the trigger should also route the payload to the agentic workflow. The alarm still keeps humans informed, but the architecture redirects machine-readable context toward the system that can analyze and act.

The preferred pattern is:

OCI Monitoring / Events / Log Analytics / APM / Database signals -> OCI Streaming or Events -> OCI Functions -> Agent workflow

Slack, email, service desk, or collaboration notifications remain important, but they are supervisory channels. The operational payload should flow into the autonomous resilience framework.

3. Agent intelligence loop

The agent intelligence loop is the core of the architecture. It turns an alarm into an operational decision.

Figure 4: A bounded agent run observes, scopes, detects, plans, governs, simulates, verifies, and reports before accepting the next trusted signal.

The loop includes several roles:

• Collector: gathers and enriches evidence from multiple OCI services.
• Detector: classifies the issue, such as performance degradation, outage, capacity pressure, database contention, security anomaly, or configuration drift.
• Context retriever: uses Oracle AI Vector Search, Select AI Agent tools, and the Resource Analytics property graph to retrieve grounded knowledge, query operational state, and identify dependent resources.
• Analytical investigator: uses deterministic queries and Oracle Machine Learning analysis to test hypotheses, identify outliers, and measure whether the condition is exceptional or part of a longer trend.
• Planner: generates one or more remediation options and explains expected impact.
• Evaluator: validates the plan before execution against policy, safety, risk, and confidence thresholds.
• Tool broker and orchestrator: coordinates governed access through Autonomous AI Database MCP Server, OCI Functions, API Gateway, Oracle Integration, service APIs, and downstream enterprise systems. Private Agent Factory can provide the agent-building and lifecycle-governance environment around these workflows.

The evaluator is the most important governance component. A good AI Ops system should not simply generate a plan and execute it. It should ask whether the plan is safe, whether the evidence supports the action, whether the blast radius is acceptable, whether a human approval is required, and whether the proposed change aligns with operational policy.

4. Controlled execution and verification

Once the evaluator approves a remediation plan, the execution layer performs the action through a controlled service boundary. In OCI, this may involve OCI Functions, Resource Manager, Database Management operations, Compute APIs, autoscaling actions, integration workflows, or runbook automation.

The framework should capture every action as evidence:

• What plan was approved?
• Which policy or evaluator approved it?
• Which OCI identity executed the action?
• What resource changed?
• What was the expected outcome?
• What telemetry changed after execution?

Post-execution verification is mandatory. The evaluator should review metrics, logs, traces, database state, and application behavior after the remediation. If the system is not back within the desired state, the loop can generate another plan, escalate to a human operator, or open a problem-management workflow.

5. Incident, problem, and knowledge management

The agent’s responsibility does not end with remediation. The final layer turns an operational event into organizational learning.

Figure 5: Operational events become reusable knowledge that improves future agent decisions.

After an incident, the framework should generate:

• An incident summary with timeline, affected systems, detected symptoms, and business impact.
• A root cause analysis that connects evidence from metrics, logs, traces, SQL data, and events.
• A remediation record describing what was proposed, what was approved, what was executed, and what changed.
• A problem-management recommendation if the issue reflects a recurring defect, capacity trend, design weakness, or process gap.
• A knowledge-base article or operational learning artifact that future agents and human operators can reuse.

This is how AI Ops becomes more than automation. Every event improves the operational memory of the enterprise.

Example: compute and database pressure

Consider a customer-facing application running on OCI Compute with an Oracle Database backend. OCI Monitoring detects that CPU utilization on the compute tier has crossed a threshold. Around the same time, Database Management shows increased sessions and wait events, APM shows higher transaction latency, and Log Analytics detects repeated timeout messages from the application.

In a traditional operations model, each signal may appear in a different console or dashboard. A human engineer has to assemble the timeline.

In the AI Ops framework, the alarm payload is routed to OCI Functions or OCI Streaming. The collector pulls related evidence from OCI Monitoring, APM, Log Analytics, Database Management, Ops Insights, and Resource Analytics. The Resource Analytics property graph identifies known dependencies around the saturated compute instance, and the metrics overlay shows which connected resources changed during the same time window. The agent can then estimate the blast radius instead of treating the CPU alarm as an isolated event.

Select AI Agent uses approved SQL and RAG tools to investigate structured database state and retrieve relevant runbooks, prior incidents, and change records through Oracle AI Vector Search. If the evidence includes documents, images, or video, an approved private or multimodal pipeline can extract and embed the content while preserving references to the source artifact. The detector classifies the condition as a possible application-saturation and database-response-time incident. The planner proposes remediation options such as scaling compute capacity, restarting a failed application component, tuning a database resource consumer group, or escalating because the confidence is too low for autonomous execution.

The evaluator checks the plan. If the action is low risk and policy-approved, the orchestrator executes it. If the action affects production capacity, database configuration, or business-critical workloads, the framework may require human approval through Slack, email, service desk, or an approval workflow.

After execution, the evaluator verifies whether CPU, latency, database waits, application errors, and transaction success rate returned to expected ranges. The framework then generates an incident summary and RCA, including the evidence chain that led to the remediation.

Human in the loop is a design principle

Autonomous resilience does not mean unsupervised change. The architecture should distinguish between informational actions, low-risk remediation, high-risk remediation, and business-impacting change.

Some actions can be fully autonomous, such as collecting additional diagnostics, enriching an incident, correlating logs, or creating an RCA draft. Some actions may be conditionally autonomous, such as restarting a stateless component or scaling a pool within an approved limit. Other actions should require explicit approval, such as changing database configuration, modifying network policy, altering security controls, or executing a remediation with uncertain blast radius.

The human-in-the-loop model should be built into the architecture from the beginning. Notifications are not only for awareness. They are a governance mechanism that keeps experts informed, captures approvals, and maintains trust.

Why this matters for customers

Customers do not adopt AI Ops because they want a new dashboard. They adopt it because cloud complexity is creating operational risk and slowing business decisions.

An OCI AI Ops framework can help customers answer higher-value questions:

• Which systems are most likely to fail or breach performance targets?
• Which applications are consuming more capacity than their business value justifies?
• Which database workloads need tuning, scaling, or architectural change?
• Which incidents are recurring because the root problem was never removed?
• Which operational risks are isolated, and which indicate a systemic pattern?
• Which remediation actions are safe enough to automate, and which still need expert approval?
• Which infrastructure and application investments will improve resilience the most?

This is where OCI Observability and Management services become strategic. OCI Monitoring, Log Analytics, Database Management, Ops Insights, APM, Resource Analytics, Events, Streaming, and Functions provide the operational evidence, topology, and workflow backbone. Oracle AI Database, AI Vector Search, Select AI Agent, Private AI Services Container, Private Agent Factory, Autonomous AI Database MCP Server, and Oracle Machine Learning Data Science Agent provide the governed context, retrieval, analytical, and agent-tooling plane. Evaluators provide safety. Human experts provide judgment and accountability.

Conclusion

AI Ops is not a replacement for DevOps. It is the next evolution of cloud operations. DevOps connected builders and operators. AI Ops connects observability, automation, AI agents, governance, and business decision-making into one operating model.

For OCI customers, the opportunity is significant. The services already used for monitoring, logging, application performance, database management, capacity planning, event routing, and serverless automation can become the evidence and execution foundation of an autonomous resilience framework. Oracle AI Database then adds the converged context plane: relational state, vectors, graph relationships, machine learning, database-native agents, private inference, and governed tool access.

The future of cloud operations will not be defined by how many alerts a team can process. It will be defined by how quickly the organization can convert telemetry into understanding, understanding into safe action, and action into durable operational knowledge.

Authors

• Royce Fu is a Master Principal Cloud Architect at Oracle on the Observability and Management Specialist team. His work focuses on OCI observability, AIOps, Oracle Database, and enterprise-scale operational intelligence.
• Adrian Birzu is Observability and Security Domain Blackbelt at Oracle EMEA CTO Office. Adrian has around 18 years of work experience, and his focus is on OCI, Observability, Multicloud and Security and provide solutions to our customers challenges.

Introduction

The hard part of deploying AI agents is not getting one impressive demo to work. The hard part is explaining what happened after the agent made a strange decision at 2:17 a.m., retried a tool five times, spent more tokens than expected, retrieved the wrong document, touched sensitive data, and still returned a confident answer.

Traditional application monitoring tells you whether an API was slow, an endpoint failed, or a database query timed out. That still matters. But agentic systems add a new layer of behavior: prompts, retrieved context, model choices, tool calls, retries, guardrails, evaluator scores, feedback, cost, and data exposure. If you cannot reconstruct those steps, you are not operating an AI system. You are watching outputs arrive and hoping the process behind them made sense.

AI agent observability is the discipline of making that behavior traceable, measurable, governable, and improvable. It is not only about troubleshooting. It is also about quality, safety, cost control, release management, privacy, and trust.

Why Agent Observability Matters

Agent observability matters because production teams need more than a final answer. They need enough evidence to understand what the agent saw, what it decided, what it did, what it cost, and whether it stayed inside the boundaries of policy and data protection.

• Debugging: Traces show causality across multi-step LLM workflows, so teams can identify the failing span and root-cause errors across prompts, retrieval, tools, guardrails, and orchestration.
• Evaluation: Prompt and completion events capture non-deterministic model behavior. Those traces can feed automated evaluations for accuracy, groundedness, format compliance, safety, and other quality characteristics. User feedback tied to specific outputs adds another signal for understanding quality and engagement.
• AI safety: GenAI telemetry, combined with specialized evaluators, helps detect unsafe outputs, bias, prompt injection, privacy violations, policy bypasses, and sensitive data exposure.
• Cost tracking: Usage metrics and token counts reveal operational cost by model, workflow, tenant, feature, prompt version, tool path, or failure path.
• Performance optimization: Latency, throughput, error rate, queue time, retry count, and tool dependency timing give agents the same production performance discipline expected from traditional applications.
• Incident response automation: Observability data can trigger operational workflows such as opening an incident, attaching the trace, notifying the owner, scaling resources, throttling risky paths, or restarting failed components. High-risk actions still need policy, approval, and audit controls.
• Version control and rollback: Tracking model versions, prompt versions, tool schema versions, retrieval indexes, and deployment versions makes it possible to see whether quality or reliability changed after a release and to roll back quickly when a new version degrades behavior.

The business case is simple: an agent that cannot be explained cannot be trusted with important work, and an agent that cannot be measured cannot be optimized.

From Final Answers To Execution Traces

For early chat applications, storing the final prompt and model response was often enough. For production agents, that view is too flat.

An agent request is a workflow. It may classify intent, rewrite the question, retrieve context, call a search index, call a business API, ask the model to choose the next step, validate the output, retry a failed tool, and finally compose the answer. The prompt is only one frame in that movie.

The useful debugging unit is the trace. A trace represents one user request or agent run from beginning to end. Inside the trace, each meaningful operation becomes a span: prompt assembly, model call, retrieval, embedding, tool call, evaluator, guardrail, retry, or handoff.

This is why LLM observability feels different from classic log search. You are not just asking, “What error did the system throw?” You are asking, “What did the agent believe, what evidence did it use, what actions did it take, and where did the behavior diverge from expectation?”

A useful production trace should capture:

• Request ID, session ID, user or tenant context, and deployment version.
• Prompt template name, prompt version, variables, and model parameters.
• Model name, model version, response status, latency, token usage, cost estimate, and retry count.
• Retrieval query, document IDs, chunk IDs, scores, filters, and source system.
• Tool name, tool schema version, arguments summary, response summary, error, retry count, and side-effect status.
• Evaluator scores, guardrail results, masking decisions, safety decisions, and user feedback.
• Final answer, failure category, owner, and incident or ticket link.

OpenTelemetry matters because AI systems do not live in isolation. An agent request may cross a web application, vector database, model gateway, tool service, workflow engine, identity provider, and enterprise system of record. OpenTelemetry GenAI semantic conventions and OpenInference point toward a more portable model for correlating AI behavior with infrastructure, application, audit, and security telemetry.

The practical rule is simple: if a human would need it during an incident review, capture it as structured trace data. If the field contains sensitive data, capture a masked or summarized version instead of pretending the field does not matter.

Cost Analysis And Cost Control For AI Agents

Cost control is one of the most important reasons to invest in agent observability. A model usage dashboard can tell you that token spend increased. It usually cannot tell you why.

AI agent cost is a workflow problem. A cost spike may come from prompt bloat, retrieval returning too many chunks, verbose tool responses, evaluator calls, repeated planning loops, retry storms, failed tool calls, oversized observations, or an expensive model being used for a step that a smaller model could handle.

The right unit of cost analysis is the trace. Then break the trace down by span:

• Planning cost: How many model calls are spent deciding what to do before any useful action happens?
• Retrieval cost: How many chunks are retrieved, how large are they, and how much context is sent to the model?
• Tool cost: Are tool responses too verbose, duplicated, or poorly shaped for the model?
• Retry cost: How much spend comes from failed calls, timeouts, malformed JSON, or rejected tool arguments?
• Evaluation cost: Which evaluator calls are required online, and which can run asynchronously or offline?
• Guardrail cost: Which safety checks are necessary on every request, and which can be triggered by risk signals?
• Failure cost: How much more expensive are failed requests than successful requests?

Cost should also be attributed by business dimension: model, prompt version, retrieval index, tenant, user segment, workflow, product feature, environment, and successful outcome. A low-cost request that produces a wrong answer is not efficient. A high-cost request that avoids a costly human escalation may be justified. The useful metric is not only cost per request. It is cost per correct, safe, useful outcome.

Practical cost controls include:

• Prompt budgets: Set maximum prompt sizes and remove instructions that no longer change behavior.
• Context budgets: Limit retrieval top-k, chunk size, and full-document stuffing. Prefer source references or summaries when the model does not need raw text.
• Model routing: Use smaller or cheaper models for classification, routing, summarization, and formatting. Reserve expensive models for steps that need deeper reasoning.
• Tool response shaping: Return only the fields the agent needs. Large JSON payloads often become hidden token waste.
• Loop and retry limits: Set clear maximums for planning loops, tool retries, evaluator retries, and repair attempts.
• Caching: Cache stable retrieval results, deterministic tool outputs, embeddings, and common classification results.
• Evaluator placement: Run lightweight checks inline and heavier evaluations asynchronously when user experience allows it.
• Budget alerts: Alert on cost per trace, cost per tenant, token growth by prompt version, retry spend, and sudden shifts in model mix.
• Sampling and retention: Keep full traces for high-risk, failed, or expensive requests. Store summarized telemetry for low-risk paths when privacy and cost require it.

The counter-intuitive lesson is that optimizing model cost usually starts outside the model. Better routing, cleaner prompts, smaller retrieval payloads, tool response shaping, masking of unnecessary sensitive fields, and loop controls often produce bigger savings than simply switching models.

Quality, Safety, And Governance

If every bad outcome is labeled “LLM error,” the observability system is not helping. Production agents need a failure taxonomy that separates the failure modes teams can actually fix.

Useful categories include:

• Input failure: ambiguous user intent, unsupported request, prompt injection attempt, malformed input.
• Prompt or version failure: wrong template, untested prompt change, missing system instruction, incompatible variables.
• Retrieval failure: no relevant documents, stale content, weak ranking, excessive context, missing access control filter.
• Model failure: hallucination, refusal, malformed JSON, unsupported tool choice, weak reasoning, unsafe output.
• Tool failure: timeout, rate limit, authentication issue, schema drift, bad arguments, rejected side effect.
• Orchestration failure: infinite loop, poor routing, bad handoff, retry storm, memory contamination.
• Evaluation failure: evaluator drift, weak test set, missing golden examples, false positive safety block.
• Security or privacy failure: prompt injection, sensitive data leakage, excessive privileges, unmasked trace payload, policy bypass.
• Cost or latency failure: token explosion, slow dependency, oversized context, excessive retries.

This taxonomy becomes the vocabulary for post-incident reviews, regression tests, release gates, model routing policies, security reviews, and rollback decisions.

Evals and observability are also converging. A bad production trace can become a regression test. A user thumbs-down can become a labeled example. A high-cost trace can become a prompt optimization candidate. A new model release can be evaluated against real historical failures before it reaches traffic.

The most mature teams will not ask only, “Did the agent fail?” They will ask, “Can we turn that failure into a reusable test so it does not come back?”

Data Masking Belongs In The Observability Design

Production traces can contain prompts, retrieved passages, tool arguments, model outputs, customer records, employee data, financial details, health information, credentials, and operational secrets. Observability data can become a second copy of sensitive business data if teams simply log everything.

That is why masking, redaction, tokenization, access control, retention, and sampling must be designed with the trace model. They are not cleanup tasks after the dashboard is built.

There are three practical masking points:

• Before the model call: Remove or tokenize sensitive values that the model does not need.
• Before tool execution: Prevent sensitive or unauthorized fields from reaching tools that should not see them.
• Before telemetry export: Mask prompt inputs, tool arguments, observations, retrieved chunks, model outputs, metadata, and evaluator payloads before they leave the application or database boundary.

The tradeoff is real: the more you mask, the less raw detail you have for debugging and evaluation. The answer is not to choose visibility or privacy. The answer is to define safe observability contracts: capture enough structure to explain behavior, but avoid persisting raw sensitive content when a masked value, hash, label, row identifier, policy decision, or source reference will do.

Reference Architecture: Production Agent Observability

A practical production architecture has five layers:

1. Instrumentation layer: Capture traces and spans from the application, orchestration framework, model gateway, retrieval layer, tools, guardrails, evaluators, and feedback UI.
2. Data protection layer: Apply masking, redaction, access control, policy checks, retention rules, and sampling before telemetry becomes another sensitive data store.
3. Telemetry pipeline: Emit structured events through SDKs, OpenTelemetry, vendor APIs, or database-native history views. Preserve request IDs, user/session IDs, tenant IDs, prompt versions, model versions, and deployment versions.
4. Analysis layer: Use an LLM observability platform for trace inspection, prompt comparison, evals, datasets, feedback, cost analysis, and release regression checks.
5. Operations layer: Correlate agent telemetry with APM, logs, metrics, audit, SIEM, database telemetry, and incident management workflows.

The architecture should answer five questions quickly:

• What happened in this request?
• Which step caused the failure, latency, cost spike, or policy block?
• Did this behavior start after a prompt, model, data, tool, policy, or code change?
• Did sensitive data move farther than it needed to?
• Can this case become a test, alert, masking rule, rollback trigger, or remediation workflow?

Tooling Landscape: Where Each Option Fits

No single tool wins every scenario. The right choice depends on whether you prioritize deep eval workflows, open-source self-hosting, framework-native tracing, gateway-level observability, enterprise production support, data-residency controls, or OpenTelemetry portability.

The most common pattern will be hybrid: use a specialized LLM observability tool for prompt, trace, eval, and feedback depth, while exporting or correlating key signals into the broader enterprise observability stack.

Balancing Efficiency And Data Security

The efficiency question is not only “Which tool gives me the best dashboard?” It is also “Where should the data be observed, reduced, masked, and stored?”

For teams that need speed, an OpenAI-compatible gateway or open-source LLM observability tool can provide fast visibility into prompts, token usage, latency, sessions, tool calls, and evals. This is the path highlighted by many practical Langfuse-style implementations: start with trace visibility, add scoring, then turn production failures into datasets and tests.

For teams that need stronger data control, the architecture should shift left. Mask before telemetry export. Keep raw customer records in governed systems. Store references, policy decisions, hashes, row IDs, and summarized observations instead of full payloads when possible. Use self-hosted observability only when the operational team is ready to manage retention, patching, access control, backups, and audit.

This is where Autonomous AI Database Select AI Agent can be a strong enterprise pattern. If the agent’s primary job is to reason over database-resident data, execute database tools, or call governed business logic, running the agent framework inside Autonomous AI Database reduces data movement and lets the agent inherit database security and audit controls. The tradeoff is that you may still want a dedicated LLM observability or evaluation platform for richer prompt experiments, dataset management, human annotation, and cross-application trace analysis. In that case, export masked or summarized telemetry rather than raw sensitive records.

In other words, ADB Select AI Agent is not simply another dashboard option. It is a placement decision: put agent execution close to governed data, then publish safe operational signals outward.

Where OCI Log Analytics And LoganAI Fit

OCI Log Analytics is especially interesting for enterprise AI operations because agent behavior does not end at the LLM boundary. Agents touch identity systems, databases, integration services, ticketing platforms, cloud APIs, network paths, and business applications. Those systems already produce logs, events, and operational signals.

For OCI customers, the pragmatic pattern is to emit structured, masked agent runtime logs into OCI Log Analytics: trace ID, span ID, user or tenant context, prompt version, model name, retrieval source, tool name, latency, token usage, error class, guardrail result, masking policy result, and final failure taxonomy. Once those fields are indexed, teams can use Log Explorer, dashboards, clusters, link visualizations, and saved searches to correlate agent behavior with OCI Audit logs, application logs, infrastructure logs, database telemetry, and security events.

LoganAI adds an AI-assisted investigation layer on top of that telemetry. Oracle documents LoganAI as the AI capability in Oracle Log Analytics for analyzing logs and log-derived data, including AI-powered explanations for single logs, multiple logs, clusters, and charts. It can also fetch metrics from OCI Monitoring using MQL and mix them with logs for analysis.

That makes LoganAI a useful operational companion to LLM tracing platforms. A dedicated LLM observability tool may tell you that a retrieval step failed or a tool call retried. OCI Log Analytics can help answer the next enterprise question: what else was happening in the environment at that time, which tenant or compartment was affected, whether related audit events occurred, whether masking policies fired, and whether similar failures are clustering elsewhere.

Hands-On Example: Trace A Codex Agent With Langfuse

Langfuse is a practical way to demonstrate LLM-focused observability because it gives teams traces, generations, tool calls, sessions, token usage, scores, datasets, and feedback workflows. For Codex specifically, the Langfuse Codex observability plugin traces agent turns, model calls, tool executions, token usage, and subagent threads to Langfuse.

Use this as a development or controlled-environment example first. The plugin can upload Codex transcript data, including prompts, assistant messages, reasoning summaries, tool-call inputs and outputs, model metadata, and token usage. Do not enable it for sessions that contain data you are not comfortable storing in Langfuse.

1. Create a Langfuse project

Create a Langfuse Cloud project or use a self-hosted Langfuse instance. Generate a public key and secret key for the project.

Choose the correct base URL for your data region, for example:

• https://cloud.langfuse.com
• https://us.cloud.langfuse.com
• https://jp.cloud.langfuse.com

2. Optional: install the Langfuse skill for Codex

The Langfuse skill gives Codex Langfuse-specific guidance and documentation workflow support.

python3 ~/.codex/skills/.system/skill-installer/scripts/install-skill-from-github.py \
  --repo langfuse/skills \
  --path skills/langfuse

Verify the skill:

ls -la ~/.codex/skills/langfuse

Expected files include:

SKILL.md
references/

3. Add the Langfuse plugin marketplace

codex plugin marketplace add langfuse/codex-observability-plugin

4. Enable the tracing plugin

Depending on your Codex build, you may install the marketplace plugin explicitly:

codex plugin add tracing@codex-observability-plugin

Then verify:

codex plugin list

Expected entry:

tracing@codex-observability-plugin  installed, enabled

5. Enable plugin hooks

Edit ~/.codex/config.toml, or use <project>/.codex/config.toml for project-specific tracing.

Put plugin_hooks = true under the [features] block:

[features]
plugin_hooks = true

Enable the plugin:

[plugins."tracing@codex-observability-plugin"]
enabled = true

6. Configure Langfuse credentials

Environment variables are usually the cleanest option for public examples and automation:

export TRACE_TO_LANGFUSE="true"
export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_BASE_URL="https://us.cloud.langfuse.com"

You can also use LANGFUSE_CODEX_PUBLIC_KEY, LANGFUSE_CODEX_SECRET_KEY, and LANGFUSE_CODEX_BASE_URL if you want Codex-specific settings that do not affect other Langfuse tools.

For a JSON config file, create ~/.codex/langfuse.json or <project>/.codex/langfuse.json:

{
  "enabled": true,
  "public_key": "pk-lf-...",
  "secret_key": "sk-lf-...",
  "base_url": "https://us.cloud.langfuse.com"
}

Lock down the file:

chmod 600 ~/.codex/langfuse.json

7. Restart Codex

Restart Codex after changing plugin or hook settings. Hook configuration is loaded when Codex starts.

8. Run a test turn and verify traces

Run a simple Codex task that does not contain sensitive data. Then open the Langfuse project and look for a new trace.

The Langfuse Home view gives a project-level overview of total traces, model cost, observations over time, and scores, so you can confirm telemetry is flowing before drilling into individual runs.

The Tracing view lists every agent run with its timestamp, name, input, output, latency, cost, and token usage, so you can scan for slow, expensive, or failed traces at a glance.

Inspect:

• Overall trace duration.
• Model name and token usage.
• Tool calls and errors.
• Failed commands or retries.
• Cost by model call or generation.
• Long prompts, large outputs, or oversized tool responses.

Opening a single Codex Turn trace shows the span tree for that run — the model call, exec_command, apply_patch, and any subagent threads — alongside the prompt, output, and per-span metadata. This is the unit where you root-cause behavior: which step ran, what it produced, and where the run diverged from expectation.

The same detail view exposes the full output and structured metadata for each span — model name, token usage, service attributes, and version tags — which is exactly the evidence you need during an incident review.

You can also run with debug and stricter failure reporting during testing:

export LANGFUSE_CODEX_DEBUG="true"
export LANGFUSE_CODEX_FAIL_ON_ERROR="true"

Turn those off for normal interactive use unless you want tracing upload errors to interrupt your workflow.

9. Use the data to control behavior and cost

Once traces are flowing, start with four dashboards or saved views:

• Cost by model and prompt version: Find expensive prompts, model upgrades, and regressions.
• Cost by trace outcome: Compare successful, failed, retried, and user-rejected requests.
• Tool and retrieval overhead: Identify tools returning too much data or retrieval steps stuffing too much context.
• Quality and safety signals: Correlate evaluator scores, user feedback, guardrail blocks, and sensitive-data masking decisions.

A cost dashboard turns those views into one place to watch spend: total cost across all traces, cost by model name and environment, and the top users, traces, and observations by cost — so a spike points you straight at the workflow that caused it.

The goal is not simply to store agent traces. The goal is to create a feedback loop: observe behavior, identify waste or risk, change prompts/tools/routing/retrieval, evaluate again, and keep the traces that prove the improvement.

Conclusion

The next phase of AI operations will not be won by teams with the prettiest dashboards. It will be won by teams that can reconstruct the full story of an agent’s behavior: what it saw, what it decided, what it did, what it cost, why it failed, what data it exposed, and how that failure becomes a better system tomorrow.

As agents move from demos into enterprise workflows, observability becomes more than troubleshooting. It becomes the control plane for trust, quality, safety, and cost.

The question is not whether your agent can answer. The question is whether you can explain how it answered when production asks for proof, whether you can afford the workflow at scale, and whether you can prove that the explanation did not create a new privacy problem.

References

• Langfuse Codex observability plugin
• Langfuse skill for Codex
• Observability for Enterprise AI Agents: An Open Source Solution
• OpenTelemetry GenAI semantic conventions
• OpenInference specification
• Braintrust documentation
• Braintrust advanced tracing: Mask sensitive data
• Langfuse documentation
• Langfuse masking of sensitive LLM data
• Langfuse self-hosted data masking
• LangSmith documentation
• LangSmith: Prevent logging of sensitive data in traces
• Helicone documentation
• Arize Phoenix
• Arize tracing configuration: Sensitive data handling
• OpenLIT documentation
• OpenLIT SDK configuration
• Oracle Autonomous AI Database Select AI
• Oracle Autonomous AI Database: About Select AI Agent
• Oracle Autonomous AI Database: Select AI Agent Concepts
• Build Your Agentic Solution using Oracle Autonomous AI Database Select AI Agent
• Oracle Data Safe overview
• OCI Log Analytics: Use LoganAI
• OCI Log Analytics: Prerequisites for LoganAI
• OWASP Top 10 for Large Language Model Applications

Authors

• Royce Fu is a Master Principal Cloud Architect at Oracle on the Observability and Management Specialist team. His work focuses on OCI observability, AIOps, Oracle Database, and enterprise-scale operational intelligence.
• Adrian Birzu is Observability and Security Domain Blackbelt at Oracle EMEA CTO Office. Adrian has around 18 years of work experience, and his focus is on OCI, Observability, Multicloud and Security and provide solutions to our customers challenges.

Introduction: The Transformation Has Already Begun

If you are a DBA, Cloud Architect, or IT operations professional, you have likely noticed a seismic shift in how technology teams operate. The runbooks you maintained, the monitoring dashboards you built, the incident response playbooks you refined over years—all of these are being reimagined through the lens of AI.

This is not about replacing your expertise. It is about amplifying it.

The emergence of AI coding assistants and agentic AI systems has created a new discipline: AI Architecture—the practice of designing, orchestrating, and governing an ecosystem of AI tools that work together to enhance your operational capabilities. Just as Cloud Architecture transformed how we think about infrastructure, AI Architecture is transforming how we think about knowledge work itself.

This post—the first in the AI Architect series—draws insights from Anthropic’s The Complete Guide to Building Skills for Claude and maps those concepts to the realities of enterprise IT. Whether you manage Oracle databases, orchestrate cloud infrastructure, or run an operations center, this guide will help you understand how to build a personal AI ecosystem that makes you dramatically more effective.

From Prompt Engineering to Execution Design

The Old Way: Prompt Engineering

Most IT professionals start their AI journey the same way: typing ad-hoc questions into ChatGPT or Claude. “How do I tune a PGA in Oracle 19c?” or “Write me a Terraform module for an OCI VCN.” This is prompt engineering—crafting one-off requests and hoping for a useful response.

It works. But it does not scale.

Every time you start a new conversation, you lose context. You re-explain your environment, your naming conventions, your compliance requirements. You copy-paste responses and manually verify them. This is the equivalent of SSH-ing into a server to run ad-hoc commands—functional, but not operational.

The New Way: Execution Design

Anthropic’s skills guide introduces a paradigm shift: moving from prompt engineering to execution design. Instead of asking an AI to do something, you design a system that enables the AI to do it consistently, correctly, and autonomously.

The core insight is this:

MCP (Model Context Protocol) gives the AI the kitchen. Skills give it the recipe.

In IT terms: MCP is like giving your AI access to APIs, CLIs, and databases. Skills are like giving it your runbooks, SOPs, and tribal knowledge—packaged in a way the AI can actually use.

This distinction is critical for IT professionals. You already have the recipes. Your runbooks, your monitoring strategies, your incident response procedures—these are the institutional knowledge that makes your team effective. The question is: how do you encode that knowledge so an AI agent can leverage it?

Understanding the AI Skills Framework

Anthropic’s guide defines a skill as a structured set of instructions, packaged in a folder, that teaches an AI how to perform specific tasks consistently. Think of it as an Infrastructure-as-Code template, but for AI behavior.

Skill Anatomy: A SKILL.md File

At the heart of every skill is a SKILL.md file—a Markdown document with YAML frontmatter that tells the AI:

• When to activate (what triggers this skill?)
• How to execute (what are the step-by-step instructions?)
• What to reference (what scripts, templates, or documentation should it consult?)

---
name: oracle-awr-analysis
description: Analyze Oracle AWR reports to identify performance bottlenecks
  and generate recommendations for database tuning.
---

## Instructions

1. Read the uploaded AWR report (HTML or text format)
2. Identify the Top 5 Wait Events by DB Time percentage
3. Cross-reference with the SQL Statistics section
4. Check the Instance Efficiency Percentages
5. Generate a summary with:
   - Root cause analysis for each major wait event
   - Specific init.ora parameter recommendations
   - SQL tuning recommendations with evidence
6. Format output as a structured report with severity levels

For someone who has spent years writing runbooks, this format should feel natural. The difference is that this runbook is not for a junior DBA to follow—it is for an AI agent to execute autonomously.

Progressive Disclosure: Loading Context On Demand

One of the most important concepts from the guide is progressive disclosure—a three-level system for loading information efficiently:

Why does this matter? Because context windows are like memory. Every token loaded into the context window is working memory consumed. If you dump 50 runbooks into a prompt, the AI’s performance degrades—just like an application with memory pressure.

Progressive disclosure is the AI equivalent of just-in-time resource allocation. Load what you need, when you need it.

For those managing large environments, this is analogous to how you architect monitoring solutions: you do not push every metric to every dashboard. You define tiers—high-level health checks always visible, detailed diagnostics available on drill-down, and raw telemetry accessible only when investigating specific incidents.

Building Your AI Ecosystem: A Practical Architecture

Now let us translate these concepts into a practical architecture that IT professionals can implement today.

Layer 1: The Foundation — AI Coding Assistants

Your AI ecosystem starts with a capable AI assistant. Today, the leading options include:

• Claude (Anthropic) — via Claude.ai, Claude Code CLI, or API
• Gemini (Google) — via Gemini CLI, Gemini Canvas, or API
• ChatGPT (OpenAI) — via ChatGPT, Codex CLI, or API
• GitHub Copilot — integrated IDE experience

These are your compute engines. Just as you would not run production workloads on a single VM, you should not rely on a single AI tool. Each has strengths:

Layer 2: The Connective Tissue — Model Context Protocol (MCP)

MCP is an open protocol that gives AI assistants access to external tools and data sources. Think of it as an API gateway for AI agents.

For IT professionals, MCP servers can connect your AI to:

• Database systems — query execution, schema exploration, performance metrics
• Cloud APIs — OCI, AWS, Azure, GCP resource management
• Monitoring platforms — metrics queries, log analysis, alert management
• Version control — Git operations, code review, change tracking
• Ticketing systems — Jira, ServiceNow ticket creation and updates

┌─────────────────────────────────────┐
│         AI Assistant (Claude)       │
│                                     │
│  ┌──────────┐  ┌──────────────┐     │
│  │  Skills   │  │   Memory     │     │
│  │ (Recipes) │  │  (Context)   │     │
│  └──────────┘  └──────────────┘     │
│                                     │
└─────────────┬───────────────────────┘
              │ MCP Protocol
    ┌─────────┼──────────┐
    │         │          │
    ▼         ▼          ▼
┌────────┐ ┌────────┐ ┌────────┐
│Database│ │ Cloud  │ │Monitor │
│ Server │ │  API   │ │  API   │
│(Oracle,│ │(OCI,   │ │(APM,   │
│ MySQL) │ │ AWS)   │ │ LA)    │
└────────┘ └────────┘ └────────┘

The power of MCP is that it transforms your AI from a conversational chatbot into an operational agent that can actually query your systems, analyze real data, and take actions.

Layer 3: The Intelligence — Skills Library

This is where your institutional knowledge lives. Your skills library is a collection of SKILL.md files organized by domain:

.agents/
├── skills/
│   ├── database/
│   │   ├── awr-analysis/
│   │   │   ├── SKILL.md
│   │   │   └── scripts/
│   │   │       └── parse_awr.py
│   │   ├── capacity-planning/
│   │   │   ├── SKILL.md
│   │   │   └── references/
│   │   │       └── sizing_guidelines.md
│   │   └── incident-response/
│   │       ├── SKILL.md
│   │       └── templates/
│   │           └── incident_report.md
│   ├── cloud/
│   │   ├── oci-deployment/
│   │   │   └── SKILL.md
│   │   └── cost-optimization/
│   │       └── SKILL.md
│   └── monitoring/
│       ├── log-analysis/
│       │   └── SKILL.md
│       └── alert-triage/
│           └── SKILL.md
└── workflows/
    ├── weekly-health-check.md
    └── incident-response.md

Each skill encodes a specific capability:

Example: Database Incident Response Skill

---
name: db-incident-response
description: Triage and diagnose Oracle database performance incidents
  using AWR, ASH, and system metrics. Generate structured incident reports.
---

## Trigger Conditions
Activate when the user reports:
- Database performance degradation
- Application timeout errors
- High wait event alerts
- ORA- errors in production

## Workflow

### Phase 1: Triage (< 5 minutes)
1. Query current active sessions: `SELECT * FROM V$SESSION WHERE STATUS='ACTIVE'`
2. Check for blocking sessions: `SELECT * FROM V$LOCK WHERE BLOCK > 0`
3. Review latest AWR snapshot for Top Wait Events
4. Classify severity: P1 (outage), P2 (degraded), P3 (warning)

### Phase 2: Diagnosis (< 15 minutes)
1. Generate ASH report for the incident window
2. Identify top SQL by elapsed time
3. Check for plan changes via DBA_HIST_SQL_PLAN
4. Review system statistics (CPU, I/O, memory)

### Phase 3: Report Generation
Use the template in templates/incident_report.md to generate:
- Timeline of events
- Root cause analysis
- Immediate actions taken
- Long-term recommendations

This is your runbook, but supercharged. The AI does not just read it—it executes it, querying your actual database through MCP, analyzing real metrics, and generating a report you can hand to management.

Layer 4: The Memory — Persistent Context

One of the most transformative capabilities in modern AI systems is persistent memory—the ability for an AI to remember context across conversations.

For IT professionals, this means:

• Environment knowledge: The AI remembers your database versions, patch levels, cluster topology, and naming conventions
• Historical context: Past incidents, performance baselines, and known issues
• Team preferences: Your coding standards, documentation templates, and approval workflows
• Project state: Where you left off on a migration, what has been tested, what remains

This is the equivalent of having a team member who never forgets the institutional knowledge that typically lives in someone’s head or scattered across wikis.

Practical implementation: AI tools like Claude support memory through knowledge items and conversation history. You can accelerate this by creating a structured context file:

# Environment Context

## Database Infrastructure
- Production: Oracle 19c RAC (2-node) on OCI BM.Standard.E4
- Standby: Active Data Guard in different region
- PDB architecture: 1 CDB, 4 PDBs (HR, FIN, SCM, CX)

## Cloud Infrastructure  
- Primary region: us-ashburn-1
- DR region: us-phoenix-1
- Compartment structure: prod/nonprod/shared-services

## Monitoring Stack
- OCI Logging Analytics for log consolidation
- OCI APM for application monitoring
- Custom metrics via OCI Monitoring API
- Alerting via OCI Notifications + PagerDuty

## Standards
- Terraform for all infrastructure provisioning
- Ansible for database patching
- Git branching: main → staging → feature/*
- Change management: ServiceNow tickets required for prod changes

The AIOps Transformation: From Reactive to Predictive

For IT operations teams, this AI ecosystem enables a fundamental shift in how you work:

Traditional IT Operations

Alert fires → Engineer investigates → Manual diagnosis → Fix applied
              (15-30 min)              (30-60 min)       (variable)

Each step requires context loading: What environment is this? What changed recently? Has this happened before? What is the standard fix?

AI-Augmented Operations (AIOps)

Alert fires → AI agent triages with full context → Diagnosis + recommendations ready
              (< 2 min)                             (< 5 min)

The AI agent already knows your environment, has access to your monitoring data via MCP, and follows your incident response skill. Your role shifts from executor to reviewer and decision-maker.

What This Looks Like in Practice

Scenario: A P2 database performance alert fires at 2 AM.

Without AI ecosystem:

1. On-call DBA receives page
2. VPNs into the environment
3. Tries to remember which database, which application
4. Runs ad-hoc diagnostic queries
5. Googles error codes
6. Eventually identifies the issue
7. Manually writes the incident report next morning

With AI ecosystem:

1. On-call DBA receives page with AI-generated triage summary
2. Summary includes: affected database, top wait events, suspect SQLs, comparison to historical baseline, and recommended actions
3. DBA reviews the analysis, confirms the diagnosis
4. AI generates the incident report with full evidence
5. Total time: 15 minutes instead of 2 hours

Getting Started: Your First 30 Days

Here is a practical roadmap for building your AI ecosystem:

Week 1: Foundation

• Set up Claude (or your preferred AI) with a professional plan
• Create your environment context file (see template above)
• Start a .agents/ directory in your team’s Git repository
• Write your first SKILL.md for a task you do frequently

Week 2: Skills Development

• Convert your top 3 most-used runbooks into skills
• Add reference files (scripts, templates, documentation)
• Test each skill by running through it with your AI assistant
• Iterate based on what the AI gets right and wrong

Week 3: MCP Integration

• Explore available MCP servers for your technology stack
• Set up at least one MCP connection (database, cloud API, or monitoring)
• Create a skill that leverages MCP for real data analysis
• Document the integration for your team

Week 4: Team Adoption

• Share your skills library with your team
• Run a workshop demonstrating the AI ecosystem
• Collect feedback and prioritize the next set of skills to build
• Establish governance: who reviews and approves skills?

Testing and Quality: The Verification Loop

Anthropic’s guide emphasizes that skills should be tested just like code. This resonates deeply with IT professionals who understand the importance of validation.

The guide recommends three levels of testing:

1. Manual Testing: Run through the skill interactively and verify the output matches expectations. This is your smoke test.

2. Scripted Testing: Create test scenarios with known inputs and expected outputs. For a database analysis skill, this might mean feeding it a sample AWR report and verifying it identifies the correct bottlenecks.

3. Metrics Monitoring: Track how often skills are triggered correctly, how many tool calls they make, and their failure rates. This is your SLA monitoring for AI operations.

The key metrics to watch:

Security and Governance: Operating AI in Enterprise Environments

For IT professionals working in regulated environments, governance is non-negotiable. Here are the key considerations:

Data Classification

• Never encode credentials, tokens, or secrets in skill files
• Use environment variables or secret management tools for sensitive configuration
• Classify your skills by the data sensitivity they handle

Access Control

• Store skills in version-controlled repositories with appropriate access controls
• Use code review processes for skill changes, just like infrastructure code
• Implement approval workflows for skills that interact with production systems

Audit and Compliance

• Log all AI agent actions, especially those involving production systems
• Maintain an audit trail of skill executions and their outcomes
• Regularly review and update skills to reflect policy changes

Responsible Use

• Always have a human review AI-generated recommendations before applying to production
• Define clear escalation paths for when AI diagnosis is uncertain
• Establish boundaries: what can the AI do autonomously vs. what requires human approval?

The Bigger Picture: Skills as a Cross-Platform Standard

One of the most compelling aspects of the skills framework is that SKILL.md is an open standard. It works across:

• Claude Code (Anthropic)
• Gemini CLI (Google)
• OpenAI Codex (OpenAI)
• Any AI assistant that can read Markdown files

This means your investment in building skills is portable. If you switch AI providers—or use multiple providers for different tasks—your skills library moves with you. This is analogous to how Terraform provides cloud-agnostic infrastructure definitions: write once, execute anywhere.

For IT organizations evaluating AI tools, this portability significantly de-risks adoption. You are not locked into a vendor; you are building a knowledge asset that appreciates over time.

Conclusion: The AI Architect Mindset

The transition from traditional IT operations to AI-augmented operations is not about learning a new tool. It is about adopting a new architecture mindset:

1. Think in systems, not prompts: Design your AI ecosystem as an architecture with layers, interfaces, and governance—not as a collection of chat conversations.

2. Encode knowledge, not just commands: Your institutional knowledge—runbooks, best practices, lessons learned—is your most valuable asset. Package it as skills so it can be leveraged consistently.

3. Design for progressive disclosure: Load context efficiently. Not everything needs to be in the prompt. Structure your knowledge so the AI can find what it needs, when it needs it.

4. Verify and iterate: Test your skills like you test your infrastructure. Monitor their effectiveness. Continuously improve them based on real-world usage.

5. Govern like production: Apply the same discipline to your AI ecosystem that you apply to your production environments—access control, change management, audit trails, and security reviews.

The AI Architect does not just use AI tools. They design the system in which AI tools operate most effectively—bringing together the right models, the right context, the right skills, and the right governance to create an ecosystem that amplifies the entire team’s capabilities.

In the next installments of this series, we will dive deeper into practical implementations: building MCP integrations for Oracle databases, creating monitoring skills for OCI, and designing AI-powered incident response workflows.

The future of IT operations is not AI replacing DBAs and Cloud Architects. It is AI making every DBA and Cloud Architect ten times more effective.

References

• The Complete Guide to Building Skills for Claude (PDF) — Anthropic
• Model Context Protocol (MCP) Documentation — Anthropic
• Building Effective Agents — Anthropic Engineering
• Claude Code Documentation — Anthropic
• OCI Observability and Management — Oracle Cloud Infrastructure

Authors: Royce Fu and Robert Brouwer, Software Development, OCI Application Performance Monitoring

Introduction

Oracle Fusion Applications represent one of the most comprehensive enterprise SaaS solutions, supporting critical business functions across Human Capital Management (HCM), Enterprise Resource Planning (ERP), Supply Chain Management (SCM), and Customer Experience (CX). As organizations increasingly rely on Fusion Cloud Apps for mission-critical operations, understanding the real user experience becomes paramount for ensuring optimal performance, identifying bottlenecks, and maintaining user satisfaction.

Oracle Cloud Infrastructure (OCI) Application Performance Monitoring (APM) Real User Monitoring (RUM) Browser Agent provides a powerful solution for gaining deep visibility into how users interact with Fusion Cloud Applications. By deploying the APM Browser Agent, organizations can monitor real user sessions, track page performance, analyze user behavior patterns, and correlate performance metrics with business outcomes—all from the browser perspective.

This blog post provides a comprehensive guide to implementing OCI APM RUM Browser Agent for Oracle Fusion SaaS Cloud Apps, covering both ADF (Application Development Framework) and Redwood page implementations, and exploring key use cases that enable data-driven performance optimization.

Reference Architecture: OCI APM Real User Monitoring for Oracle Fusion SaaS Cloud Apps

OCI APM Real User Monitoring captures user interaction data directly from the browser, providing comprehensive insights into the actual user experience. The APM Browser Agent records:

• Page Load Performance: Complete page load times, resource loading, and rendering metrics
• User Interactions: Clicks, navigation patterns, and user session flows
• AJAX Calls: API call performance and error rates
• JavaScript Errors: Client-side errors that impact user experience
• Geographic Data: User location-based performance analysis
• Device Information: Browser, device type, and connection characteristics
• Application Performance Index (Apdex): Standardized performance scoring

The APM Browser Agent sends browser spans and RUM metrics to OCI APM, where they can be visualized in Trace Explorer and Metrics Explorer, enabling comprehensive analysis of user experience across Fusion Cloud Applications. Beyond deep-dive analysis, this telemetry allows you to utilize out-of-the-box RUM Dashboards and define Custom Metrics to detect and alert on Fusion environment performance issues proactively.

Figure 1: OCI APM Real User Monitoring for Oracle Fusion SaaS Cloud Apps

Prerequisites

Before implementing APM RUM for Fusion Cloud Apps, ensure you have:

1. OCI APM Domain: An active APM domain in your OCI tenancy
   • Follow these quick steps to create a domain:
     • Sign in to the Oracle Cloud Infrastructure console.
     • Open the navigation menu, click Observability & Management, and under Application Performance Monitoring, click Administration.
     • Select your target compartment, click Create APM Domain, and fill in the required details (Name, Compartment, Description, etc.).
     • Click Create APM domain and wait for the domain to be provisioned.
   • For full setup details, see Create an APM Domain.
2. APM Data Key: The public data key from your APM domain
   • Navigate to APM Domain > Resources > Data Keys
   • Copy the Public Data Key
3. Data Upload Endpoint: The OCI data upload endpoint URL for your APM domain
   • Format: https://<domain-id>.apm-agt.<region>.oci.oraclecloud.com
   • Found in APM Domain > Administration > Data Upload Endpoint
   • Copy the Data Upload Endpoint
   • Figure 2: OCI APM Data Key and Data Upload Endpoint

4. ADF Sandbox Access: Ability to create and publish sandboxes in Fusion Applications

5. Redwood Visual Builder Studio project access: Ability to export, import, preview, and publish an application-extension workspace for Fusion Redwood pages.
   • Note: The Fusion role alone does not grant access to VB Studio — the user also needs a VB Studio role in the Identity Domain.
   • If role synchronization is enabled (the common case), assign one of:
     • ORA_SYNC_ENABLED_ADMINISTRATOR_ABSTRACT — maps to VB Studio Administrator
     • ORA_SYNC_ENABLED_DEVELOPER_ABSTRACT — maps to VB Studio Developer (These are assigned in the Fusion Security Console and auto-sync to the Identity Domain.)
   • If role synchronization is disabled, you must manually assign in the Identity Domain console:
     • DEVELOPER_ADMINISTRATOR — VB Studio admin access (create projects, manage settings)
     • DEVELOPER_USER — VB Studio developer access (create/develop projects).

Implementation: Deploying APM Browser Agent

Oracle Fusion Applications consist of two distinct page architectures, each requiring a specific implementation approach:

• ADF Pages: Legacy pages identified by /faces/ in the URL
• Redwood Pages: Modern pages identified by /redwood/ in the URL

For complete coverage, you must implement the APM Browser Agent for both page types.

Part 1: Adding RUM Agent to Fusion ADF Pages

ADF pages use a global page template that can be customized to include the APM Browser Agent across all ADF pages in your Fusion instance.

Step 1: Access Global Page Template Editor

1. Log in to your Fusion Cloud Application
2. Navigate to Settings > Actions > Edit Global Page Template
   • Figure 3: Edit Global Page Template
3. Create or activate a sandbox:
   • Click Activate a Sandbox
   • Figure 4: Activate a Sandbox
   • Select Page Template Composer option
   • Click Create and Enter
   • Enter the sandbox by clicking on its name
   • Figure 5: Create Sandbox

Step 2: Edit the Template

1. Click Edit Global Page Template from the menu (now that you’re in a sandbox)
   • Figure 6: Edit Global Page Template
2. Ensure the page is in Design mode

   • When active, you’ll see a bar with ‘Add content’

     ‘Select’

     ‘Structure’

   • Figure 7: Design Mode
3. Select Structure from that bar
4. Click the ”+” symbol at the bottom to add a new HTML markup component
   • Figure 8: Add Component
5. Select Component and click Add to add the HTML Markup, then Close the popup
   • Figure 9: Add Component
   • Figure 10: Add Component
6. Switch to Add Content tab and click the gear setting button of the HTML Markup component
7. Open the Expression by clicking the settings icon
   • Figure 11: Open Expression
8. Click Expression Builder
   • Figure 12: Expression Builder

Step 3: Add APM Browser Agent Script

In the Expression Builder, add the following HTML markup:

<!-- APM RUM Library -->
<!-- Note: static.oracle.com requires maintenance to update the version -->
<!-- Alternative: Use region-specific script if needed -->
<script type="text/javascript" crossorigin="anonymous" src="https://static.oracle.com/cdn/apm/oci/1.1/1.1-1xx/1.1-148/apmrum.fus.base.combi.min.js"></script>

<script type="text/javascript">
function initializeAPMRumConf() {
  const apmRumConfig = document.apmrum || window.apmrum || (document.apmrum = {});
  apmRumConfig.custEP = apmRumConfig.custEP || {};
  
  /* CUSTOMER SPECIFIC VALUES */
  apmRumConfig.custEP.OracleAPMPublicDataKey = 'YOUR_PUBLIC_DATA_KEY';
  apmRumConfig.custEP.ociDataUploadEndpoint = 'https://YOUR_DOMAIN_ID.apm-agt.us-phoenix-1.oci.oraclecloud.com';
  apmRumConfig.custEP.serviceName = 'Fusion';
  apmRumConfig.custEP.webApplication = 'ADF-Page';
  apmRumConfig.traceSupportingEndpoints = apmRumConfig.traceSupportingEndpoints || [
    { headers: [], hostPattern: '.*' }
  ];
  
  /* Uncomment the lines below only after the required privacy review. */
  // apmRumConfig.custEP.trackPlainUserId = true;
  // apmRumConfig.trackPlainUserId = true;
  // apmRumConfig.trackScreenText = true;
  
  /* END OF CUSTOMER SPECIFIC SETTINGS */
  apmRumConfig.confInst = (apmRumConfig.confInst || new Set()).add('custEP');
}
initializeAPMRumConf();
</script>

Configuration Notes:

• Replace YOUR_PUBLIC_DATA_KEY with your APM domain’s public data key
• Replace YOUR_DOMAIN_ID.apm-agt.us-phoenix-1.oci.oraclecloud.com with your actual data upload endpoint
• Adjust the region in the endpoint URL to match your APM domain region
• The webApplication value is set to 'ADF-Page' to identify ADF pages in APM
• Screen-text collection and plain user-name reporting remain disabled unless you explicitly enable them after the required privacy and security review

User ID Hashing:

• By default, APM hashes user IDs to protect privacy
• To track plain text user IDs (for debugging), uncomment: apmRumConfig.custEP.trackPlainUserId = true;
• Warning: Only enable plain text user IDs if required for troubleshooting and ensure compliance with privacy regulations

Personally Identifiable Information (PII):

• Review APM documentation on Hiding Personally Identifiable Information
• Configure PII masking if your application contains sensitive user data
• Ensure compliance with GDPR, CCPA, and other privacy regulations

Set the Content-Security-Policy Header

The browser must be permitted to load the APM RUM library from the location used in the script. Add either static.oracle.com or your region-specific ingest-rum.apm-agt.<region>.oci.oraclecloud.com host to the applicable ADF Content-Security-Policy ALLOWED_ORIGINS configuration. Preserve any origins that are already configured. The APM data upload endpoint must also be permitted for outbound browser connections. See ADF Faces Configuration and Content Security Policy Error Correction.

Step 4: Publish the Changes

1. Click Apply or Save to close the popup
2. Click Publish
3. Click Yes in the confirmation dialog
   • Figure 13: Publish

The APM Browser Agent is now deployed to all ADF pages in your Fusion instance.

Part 2: Adding RUM Agent to Fusion Redwood Pages

Starting from Oracle Fusion Release 26C, the earlier approach of adding JavaScript to the Unified Module in Advanced mode is no longer used. Instead, the Fusion APM RUM Assistant transforms a current Visual Builder workspace export into an importable ZIP, embeds the browser agent and shared APM configuration, and activates the module in the eligible Redwood App UI overrides selected for that extension. The administrator does not need to edit JavaScript or JSON.

Access: To request access to the Fusion APM RUM Assistant, contact your Oracle account team so they can engage the development team, or open a My Oracle Support service request for additional support.

Deployment boundary: Generate and review the ZIP locally, then import it into a non-production Visual Builder workspace first. Treat Preview, RUM upload verification, and publication as separate administrator approval gates. Keep the original workspace export as the rollback artifact.

Step 1: Open the Current Workspace from a Redwood Page

1. Navigate to any Redwood page in your Fusion application. For example, navigate to Me > Journeys, or another Redwood page you know.

   Figure 14: Redwood page

2. Click the Profile icon in the upper-right corner.

3. Select Edit the Page in Visual Builder Studio.

   Figure 15: Edit the page in Visual Builder Studio

   The first time you access this option, Fusion automatically creates a Git repository for your Oracle Application or App Extension project. This is a one-time activity.

   Figure 16: Git repository

Step 2: Export the Current Workspace

1. The project Git repository opens and displays the page template.

   Figure 17: Page template

2. From the menu in the upper-right corner, select Export. The downloaded archive contains the current files from the workspace branch.

   Figure 18: Export and Import are available from the Visual Builder workspace menu

3. Keep this original ZIP unchanged as the rollback copy. Run the assistant against a copy of this current export, not against an older generated APM ZIP.

For more information, see Export Your Workspace as an Archive.

Step 3: Prepare and Start the Assistant

1. Extract fusion-apm-rum-assistant.zip into a private working folder.
2. Copy the Fusion workspace export into the assistant’s downloads folder.
3. Start the assistant:
   • macOS: Double-click Fusion APM RUM Assistant.command.
   • Windows: Double-click Fusion APM RUM Assistant.bat.
4. If prompted that Python 3 is unavailable, ask your local IT team to install Python 3 and then start the assistant again.

Figure 19: Fusion APM RUM Assistant working folder on macOS; environment-specific archive names are redacted

On the first run, enter the following values from the target OCI APM domain:

• OCI region: The region containing the APM domain. The assistant uses this region to obtain the browser agent while generating the ZIP.
• APM public data key: Found under APM domain details > Data Keys > Public Data Key.
• APM data upload endpoint: Shown in the APM domain details. Copy it directly and confirm that it is the HTTPS apm-agt endpoint for the intended APM domain and OCI realm; do not substitute a general-purpose HTTPS URL. This assistant version checks the URL format but does not prove that the host belongs to the intended APM domain, so this manual comparison is required.

The local configuration is stored in the extracted assistant folder for reuse. The generated ZIP also contains deployment-specific APM settings, so keep both the working folder and generated ZIP within the approved deployment process. The assistant writes trackScreenText: false and trackPlainUserId: false in its customer configuration. Static inspection of this package also found that the embedded Fusion base library initializes a separate fusBaseConfig.trackPlainUserId value to true. Treat the effective user-name behavior as unresolved until the target APM data confirms that identity is hashed or omitted, and do not publish if plain identifiers appear.

Step 4: Select the Fusion App UI Scope

The assistant asks two scope questions. Both default to No:

1. Refresh the stored App UI catalog from Fusion?
   • Choose No to use the saved catalog, or when you only want to instrument App UIs already represented in the exported workspace.
   • Choose Yes for first-time pillar expansion or after Fusion changes. Enter the Fusion pod host name when prompted. The assistant downloads the pod’s current app-ui-info catalog and saves it beside the local APM configuration.
   • If the pod returns an interactive sign-in page instead of the catalog, download the catalog URL through an authenticated Fusion browser session, save it as config/app-ui-info.json, and run the assistant again without refreshing it.
2. Add all eligible App UIs in one Fusion pillar?
   • Choose No to use only the App UIs already present in the export.
   • Choose Yes, then enter exactly one pillar: hcm, cx, erp, or scm. A comma-separated list is not accepted.

Each generated ZIP covers one selected pillar. To instrument another pillar, start from that pillar’s corresponding extension/workspace export and create a separate output; rerunning against the same original export does not combine pillars.

Pillar expansion includes only applications that the catalog explicitly identifies as extensible. In this mode, the assistant deliberately excludes unified-module, creates the selected App UI override directories, merges the required provider dependencies, and applies the shared APM RUM module import. When you choose No for pillar expansion, the assistant applies the import to every App UI directory already present in the export. Because this package version does not remove an existing unified-module directory in that mode, confirm that the source workspace does not contain a unified-module App UI override before proceeding.

If the downloads folder contains several Fusion export ZIPs, select the intended archive from the numbered list.

Review the Generation Summary

Do not import the generated ZIP until the summary matches the intended scope. Review these lines in particular:

• Extension and Manifest: Confirm the expected application extension and export format.
• Expansion pillar: Confirm the one intended business pillar.
• Catalog applications selected: Confirm that eligible App UIs were found.
• Applications created and Applications updated: At least one of these values must be greater than zero. If both are zero, the ZIP contains APM resources but does not activate RUM in any App UI; correct the export or select the confirmed pillar and rerun the assistant.
• Created: Record the exact output ZIP that will be imported.

In one validated HCM packaging run, the current pod catalog contained 199 HCM applications: 174 were marked extensible and added, 25 were skipped, 93 provider dependencies were added, and unified-module was excluded. These counts describe that pod and release only; expect them to change across Fusion environments and updates.

Figure 20: Successful HCM pillar-expansion packaging summary; environment-specific identifiers and local paths are redacted

Validation scope: This result validates local macOS packaging and the structure of that generated HCM ZIP. It does not by itself validate Windows execution, Visual Builder import or build, live RUM delivery, privacy behavior, or CSP. Complete the non-production checks in Step 5 before publishing. The displayed Agent source is the packaging-time download source; the generated extension embeds that library for runtime use.

Step 5: Import, Preview, Verify, and Publish

1. Return to the same Visual Builder workspace used for the export.
2. Confirm that no one has changed the workspace since the source export was created. If it has changed, export a fresh rollback ZIP, rerun the assistant against that archive, and review the new output before importing.
3. From the upper-right menu, select Import and choose the *-with-apm-rum.zip file from the assistant’s output folder.

4. When replacing the workspace from this complete generated export, select Delete existing files and resources to prevent duplication, then click Import. See Import Resources to Your Workspace.

   Figure 21: Import the generated APM RUM ZIP

5. Review the import result and build messages. A large pillar-wide extension can require additional processing time; the published Fusion runtime remains unchanged until you select Publish.

6. Preview representative Redwood user workflows in the selected App UIs to generate RUM activity. Do not publish yet.

   Figure 22: Preview a representative Redwood user workflow before publishing

7. Wait 5–10 minutes, then open the browser Developer Tools Network tab. Filter for apm, or more narrowly for observations/public-span, and confirm that the APM data upload endpoint returns HTTP 200 or 204.

   Figure 23: Verify the APM upload response in the browser Network tab

8. In the intended APM domain, confirm the new activity in Trace Explorer or the Real User Monitoring dashboard. Verify the expected ServiceName and WebApplication, and confirm that privacy-sensitive values are not reported.
9. Click Publish only after the preview and APM checks succeed. This deploys the changes to the eligible Redwood App UIs included in the imported extension.

The generated extension embeds the browser-agent library in its APM module. At runtime, the App UI should therefore not fetch the library from static.oracle.com or an ingest-rum host. The browser still uploads RUM observations to the configured APM data upload endpoint, which must be allowed by the application’s Content-Security-Policy connect-src rules. A CSP error in the browser console or a missing 200/204 upload response indicates that the connection policy or network path still needs attention.

If validation fails, do not publish. Retain the assistant configuration, catalog, source export, generated ZIP, and validation result as the deployment record; use the untouched source export to restore the workspace if rollback is required.

Part 3: Configure APM Domain Enrichment Rules

For complete Fusion coverage, enable the Oracle-provided Fusion (FUS) span rule to add Fusion-specific attributes and dimensions to your traces.

Enable Fusion Enrichment Rule

1. Navigate to your OCI APM Domain
2. Go to Administration > Span Rules
3. Click Create Span Enrichment
   • Figure 24: Create Span Enrichment
4. Select Oracle Provided span rules
5. Find and add the “FUS” rule
   • Figure 25: Add Span Enrichment Rule

This enrichment rule creates the following Fusion-specific attributes:

• RwApplication: Identifies the Redwood application
• RwPath: Captures the Redwood page path
• RwContainer: Identifies the Redwood container
• FusFamily: Describes the Fusion area where activity occurred (e.g., HCM, ERP, SCM, CX)

These attributes enable powerful filtering and analysis capabilities in APM Trace Explorer and Metrics Explorer.

Part 4: Verify Application Performance Monitoring Browser Agent Deployment

To generate some monitoring data and verify the setup, use the application you are configuring:

• Generate Data: Use the application you are setting up the monitoring for to generate initial monitoring data.
• View in APM: Go to the APM Trace explorer page to see the data loaded. For example, see APM screenshots below.
• Check privacy settings: For ADF, leave the plain-user-name and screen-text lines commented unless their use has completed privacy and security review. For Redwood, do not infer the effective setting from the generated customer configuration alone because the embedded Fusion base library has its own user-name setting. Confirm that the resulting APM data does not expose plain user names or screen text before publishing.

If needed, verify that the APM data upload url (window.apmrum.ociDataUploadEndpoint) is reachable by doing the following:

• Go to your application and use the Developer Tools in your browser to go to the Developer console. In most browsers, you can start the Developer Console by pressing the F12 key.
• Select the Network tab and filter the URL list by observations/public-span, if required.

• The Collector URL should be displayed with the 200 or 204 HTTP response code. If you do not see a 200 or 204 HTTP response code, then the browser is not able to access the Collector URL.

• Figure 26: Network Tab

Utilize the APM Real User Monitoring Dashboard to monitor user behavior and application performance

OCI Application Performance Monitoring provides an out-of-the-box Real User Monitoring (RUM) Dashboard that offers immediate, high-level visibility into your Fusion Cloud Apps performance without requiring any custom queries.

To access the RUM Dashboard:

1. Navigate to Observability & Management in the Oracle Cloud Console.
2. Navigate to Application Performance Monitoring, and click Real User Monitoring.
3. Use the global filters at the top to select your compartment, APM domain, Web Application and the desired time frame.

The RUM Dashboard visualizes critical performance metrics across several key dimensions automatically:

• Apdex Score: Monitor the overall satisfaction of your users in a single aggregate performance score.
• Page Performance: Track the exact, real-world time it takes for your Redwood and ADF pages to render for users.
• Page Views & Sessions: View user engagement trends and active sessions over time to understand peak traffic periods.
• Geographic Breakdown: Visually analyze which regions and countries your users are connecting from, and identify location-specific performance latency.
• Client & Browser Breakdown: Understand the device types, operating systems, and browsers your users depend on to access Fusion.
• Top Slowest Pages: Instantly pinpoint the specific Fusion pages currently exhibiting the poorest performance to prioritize code and network optimizations.

This built-in dashboard serves as an excellent starting point for everyday monitoring, performance reporting, and quickly diagnosing macro-level user experience issues without ever looking at the raw telemetry data.

Figure 27: RUM Dashboard

Gain additional insights from RUM Data

Once the APM Browser Agent is deployed and the Fusion span enrichment rules are configured, you can leverage the newly populated attributes (such as RwApplication, RwContainer, RwPath, FusFamily, FusProductCode, FusComponentType, etc.) to gain deeper, custom insights from your RUM data across various use cases:

Use Case 1: Fusion Cloud Apps Pages Clicks by User

Understand which Fusion pages users access most frequently and identify user navigation patterns.

For Redwood Pages:

show spans *  group by RwApplication,RwContainer,RwPath

Figure 28: Redwood Pages Analysis

For ADF Pages:

show spans *  group by FusFamily,FusProductCode,FusComponentType,FusContextRoot,FusRegionId

Figure 29: ADF Pages Analysis

Use Case 2: User Session Performance Profiling

Analyze the sequence of events and performance bottlenecks within individual user sessions.

show (traces) * where SessionId='<your_session_id>' order by TraceRootSpanStartTime asc

Figure 30: User Session Analysis

Use Case 3: Comprehensive Application Usage and Engagement

Analyze user engagement across both ADF modules and Redwood pages by aggregating the number of unique sessions, users, and total pageviews for each application area, ordered by highest session volume.

SHOW traces  FusFamily ||'-'|| FusProduct as "baseDetails",RwApplication ||'-'|| RwContainer ||'-'||RwPath as RedwoodDetails, count_distinct(sessionId) as sessions, count_distinct(username) as users, sum(pageviews) as activities
   where RwApplication is not omitted or FusFamily is not omitted
   group by FusFamily ||'-'|| FusProduct,RwApplication ||'-'|| RwContainer ||'-'||RwPath
order by count_distinct(sessionId) desc

Figure 31: Application Usage Analysis

APM Custom Metrics and Alarms

Beyond passively monitoring dashboards, you can configure OCI APM to proactively alert you when Fusion web pages exceed acceptable performance thresholds. This involves creating a Custom Metric to aggregate performance data based on specific criteria, and then configuring OCI Alarms.

Step 1: Create a Custom Metric

First, define a custom metric to continuously capture specific performance behavior, such as slow page loads.

1. Navigate to your APM Domain > Resources > Custom metrics.
2. Click Create custom metric.
3. Enter a Metric name (e.g., SlowFusionRedwoodPages).
4. In the Span filter box, configure the query to isolate the targeted behavior. For example:

   ServiceName = 'Fusion' and WebApplication = 'RedwoodPage'
   

5. From the Metric value source dropdown list, select the numeric metric to capture (e.g., SpanDuration).

   Figure 32: Custom Metric Value Source

6. (Optional) Select one or multiple Dimensions for fine-grained grouping (e.g., RwContainer, RwApplication, FusFamily).

   Note: if you choose too many dimensions, the custom metric will be aggregated for each combination of dimensions, which may result in high cardinality which could potentially overload monitoring service. Please reduce the number of dimensions to avoid this issue. Figure 33: Custom Metric Large Volume Warning

7. Click Next to choose metric storage options.
8. Choose your target Compartment and Namespace. The default namespace is oracle_apm_custom (note: using a non-default custom namespace requires configuring OCI identity policies).

9. Click Create. APM will now periodically aggregate and push this custom metric to the OCI Monitoring service.

   Figure 34: Custom Metric

Step 2: Configure an OCI Alarm

Finally, set up an alarm to notify your team when the custom metric breaches a threshold.

1. In the Oracle Cloud Console, navigate to Observability & Management > Monitoring > Metrics Explorer.
2. Select the compartment, metric namespace (oracle_apm_custom) and the custom metric name (e.g., SlowFusionRedwoodPages).
3. Click Create Alarm.
4. Provide Alarm Name, Alarm Summary, Metric Namespace(e.g., oracle_apm_custom), Metric Name(e.g., SlowFusionRedwoodPages), Interval(1 minute), Statistic(Mean).
5. Define the Trigger Rule (e.g., trigger when the count > 1000 within a 1-minute interval).
6. Under Destinations, select or create an OCI Notification Topic (e.g., Email, Slack/PagerDuty integration) to route the alert to your support team.
7. Click Save Alarm.

Your team will now receive automated alerts whenever Fusion page performance degrades beyond your specified thresholds!

Conclusion

Implementing OCI APM Real User Monitoring for Oracle Fusion SaaS Cloud Apps provides organizations with comprehensive visibility into real user experience, enabling data-driven performance optimization and improved user satisfaction. By deploying the APM Browser Agent for both ADF and Redwood pages, organizations can:

• Track User Behavior: Understand which pages users access and how they navigate through Fusion Apps
• Profile Performance: Analyze individual user sessions to identify performance bottlenecks
• Geographic Analysis: Understand how user location impacts application performance
• Standardized Metrics: Use Apdex scores to measure and track overall application performance
• Trend Analysis: Monitor performance over time to detect degradation and measure improvements
• Device Profiling: Optimize application performance for different devices, browsers, and connection types

The implementation uses an ADF page-template configuration for ADF pages and an assistant-generated Visual Builder workspace import for eligible Redwood App UIs. After the Fusion enrichment rule is enabled and the deployment is validated, the APM Browser Agent collects RUM data without requiring additional server-side agents.

By leveraging the insights provided by APM RUM, organizations can proactively identify and resolve performance issues, optimize user experience, and ensure that Fusion Cloud Apps deliver the performance and reliability that users expect.

References

• Real User Monitoring for Oracle Fusion SaaS Cloud Apps using OCI APM Browser Agent
• Observability and Management Solution: Oracle Fusion Applications and OCI Observability Services
• Configure APM Browser Agent for Real User Monitoring
• Deploy the APM Browser Agent With Your Application
• Configure Attributes for Real User Monitoring
• Configure Session Tracking
• Configure Ajax Calls Instrumentation
• Configure Span Rules
• Hide Personally Identifiable Information
• Verify APM Browser Agent Deployment
• Export Your Workspace as an Archive
• Import Resources to Your Workspace

Introduction

When running workloads in Oracle Kubernetes Engine (OKE) clusters, having detailed visibility into system performance is crucial for troubleshooting issues, optimizing resources, and ensuring application reliability. While Kubernetes provides basic monitoring capabilities, capturing granular process-level metrics often requires specialized tools.

In this blog post, we’ll explore how to use nmon (Nigel’s Monitor) to capture detailed process information in OKE clusters. We’ll cover both production monitoring configurations (30-second intervals) and high-resolution troubleshooting setups (1-second intervals) for capturing granular metrics when investigating performance issues.

What is nmon?

nmon is a powerful system performance monitoring tool originally developed by IBM for AIX and later ported to Linux. It provides comprehensive insights into various system aspects:

• CPU utilization by core
• Memory usage
• Disk I/O activity
• Network throughput
• Top processes by resource usage
• File system statistics
• Kernel statistics

What makes nmon particularly valuable is its ability to monitor multiple system resources simultaneously, providing a consolidated view that helps identify correlations between different performance aspects. For example, you might notice that high CPU usage coincides with specific disk I/O patterns, pointing to potential bottlenecks.

Why Use nmon in OKE Clusters?

Kubernetes environments add layers of abstraction that can make performance troubleshooting challenging. Standard monitoring solutions often aggregate metrics over longer intervals (typically 15-60 seconds), which can mask short-duration spikes that impact application performance.

Key benefits of using nmon in OKE clusters include:

1. Flexible monitoring resolution: Capture metrics at configurable intervals (30-60 seconds for production, 1 second for troubleshooting)
2. Process-level visibility: Track specific processes and their resource consumption with the -t flag
3. Low overhead: nmon is designed to be lightweight and consume minimal resources, especially at longer intervals
4. Data persistence: Record performance data for offline analysis and historical comparison
5. Visualization capabilities: Generate charts for easier interpretation of complex performance patterns
6. Dynamic configuration: Switch between production and troubleshooting modes based on operational needs

Deployment Options for nmon in OKE

There are several approaches to deploying nmon in an OKE cluster. We’ll explore two primary methods:

1. Standalone installation on individual nodes
2. Containerized deployment using DaemonSets

Option 1: Standalone Installation

For direct installation on OKE nodes, you can use the following commands:

# Download nmon and nmonchart
curl -LO http://sourceforge.net/projects/nmon/files/nmonchart40.tar

# Create directory and extract
mkdir nmonchart
tar xvf nmonchart40.tar -C nmonchart

# Install ksh (required for nmonchart)
dnf install ksh

# Run nmon with 1-second interval
nmon -f -t -s 1 -c 3600  # Captures data for 1 hour with 1-second intervals

Option 2: Containerized Deployment (Recommended)

For Kubernetes environments, a containerized approach using DaemonSets provides better manageability and integration with the cluster. This ensures nmon runs on every node without manual installation.

Step 1: Create a Dockerfile for nmon

# Dockerfile
FROM oraclelinux:8

# Install EPEL
RUN yum install -y oracle-epel-release-el8.x86_64 && \
    yum install -y yum-utils && \
    yum-config-manager --enable ol8_baseos_latest ol8_appstream ol8_addons ol8_developer_EPEL

# Install nmon
RUN yum install -y nmon && \
    yum clean all

# Create a directory for nmon logs
RUN mkdir -p /var/log/nmon
RUN mkdir -p /opt/nmon/output

# Default command with 30-second interval for production monitoring
# Override with command in DaemonSet YAML for troubleshooting scenarios
ENTRYPOINT ["nmon", "-f", "-t", "-s", "30", "-c", "2880"]

Step 2: Build and Push the Docker Image

# Build the image
docker build -t nmon:oraclelinux8 .

# Tag the image for your registry
docker tag nmon:oraclelinux8 <your-registry>/nmon-repo:v1

# Log in to your registry
docker login -u '<username>' <your-registry>

# Push the image
docker push <your-registry>/nmon-repo:v1

Replace <your-registry>, <username> with your specific Oracle Cloud Infrastructure Registry details.

Step 3: Deploy nmon as a DaemonSet

Create a YAML file named nmon-daemonset.yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nmon-daemonset
  namespace: monitoring  # Adjust as needed
  labels:
    k8s-app: nmon
spec:
  selector:
    matchLabels:
      name: nmon
  template:
    metadata:
      labels:
        name: nmon
    spec:
      # Tolerations to allow running on control-plane nodes if desired
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule

      # Enable to see host processes
      hostPID: true
      containers:
      - name: nmon
        image: <your-registry>/nmon-repo:v1
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: nmon-output
          mountPath: /opt/nmon/output
        - name: varlog
          mountPath: /var/log
        securityContext:
          privileged: true  # Required to access host metrics
      terminationGracePeriodSeconds: 30
      volumes:
      - name: nmon-output
        hostPath:
          path: /var/log/nmon
      - name: varlog
        hostPath:
          path: /var/log

Apply the DaemonSet configuration:

kubectl apply -f nmon-daemonset.yaml

Verify that the DaemonSet pods are running:

kubectl get pods -n monitoring

Configuring nmon for High-Resolution Metrics When Needed

nmon provides flexible configuration options that allow you to balance monitoring overhead with the level of detail needed for troubleshooting. The key to effective monitoring is understanding when to use different sampling intervals and which flags to enable.

Understanding nmon Command-Line Options

The core nmon parameters for data collection are:

• -f: Write to a file instead of displaying interactively
• -t: Critical flag - Include top processes in the output (required for process-level troubleshooting)
• -s <seconds>: Sampling interval (how often to collect data)
• -c <count>: Number of samples to collect
• -p: Add extra process details including command arguments
• -T: Include top threads (helps identify specific threads causing high CPU usage)

Production Monitoring vs. High-Resolution Troubleshooting

For Production Monitoring (Continuous Operation):

Use longer intervals to minimize overhead while maintaining visibility:

# 30-second intervals for production monitoring (recommended)
nmon -f -t -s 30 -c 2880  # Captures 24 hours of data at 30-second intervals

# Alternative: 60-second intervals for lower overhead
nmon -f -t -s 60 -c 1440  # Captures 24 hours of data at 60-second intervals

For High-Resolution Troubleshooting (When Issues Occur):

When investigating performance issues, switch to high-resolution monitoring to capture granular details:

# 1-second intervals for detailed troubleshooting
nmon -f -t -s 1 -c 3600  # Captures 1 hour of data at 1-second intervals

# For very short incidents (5-10 minutes)
nmon -f -t -s 1 -c 600   # Captures 10 minutes of data at 1-second intervals

Critical Configuration: The -t Flag

Always include the -t flag when deploying nmon for troubleshooting. This flag is essential for capturing top processes, which is critical for identifying which processes are consuming resources during performance incidents.

Without -t, you’ll only get system-level metrics (CPU, memory, disk) but won’t be able to identify the specific processes causing issues. This is a common oversight that can significantly limit troubleshooting effectiveness.

Process-Level Visualizations (Requires -t Flag)

When you use the -t flag, nmonchart generates process-level visualizations that are essential for troubleshooting:

Figure 6: Top Process Commands by CPU percentage - critical for identifying which processes consume CPU resources. This chart clearly shows processes like setroubleshootd, prometheus, and ruby consuming CPU at specific times. This visualization is only available when using the -t flag - without it, you cannot identify which processes are causing performance issues.

Figure 7: Top 20 processes correlation chart showing relationships between CPU seconds, Character I/O, and Memory Size. Bubble size represents memory consumption. This helps identify processes with high resource usage across multiple dimensions. Also requires the -t flag.

Critical Note: Without the -t flag, Figures 6 and 7 would not be available, making it impossible to identify which specific processes are causing performance issues. Always include -t when deploying nmon for troubleshooting.

Enhanced Troubleshooting Configuration

For comprehensive troubleshooting during critical incidents, use additional flags to capture maximum detail:

# Maximum detail configuration for troubleshooting
nmon -f -t -p -T -s 1 -c 1800  # 30 minutes of 1-second data with process and thread details

# Parameters explained:
# -f: File output
# -t: Top processes (REQUIRED for process identification)
# -p: Extra process details including command arguments
# -T: Top threads (identify specific threads causing high CPU)
# -s 1: 1-second sampling interval
# -c 1800: 1800 samples (30 minutes)

Dynamic Configuration Switching

In production environments, you can run nmon with normal intervals and dynamically switch to high-resolution mode when issues are detected:

Scenario: Intermittent CPU Spikes

1. Normal Operation: Run nmon with 30-second intervals

   nmon -f -t -s 30 -c 2880 -F /var/log/nmon/production_$(date +%y%m%d_%H%M%S).nmon
   

2. When Alert Detected: Immediately start high-resolution capture

   # Execute in the nmon container/pod
   kubectl exec -n monitoring <nmon-pod-name> -- nmon -f -t -p -T -s 1 -c 300 -F /var/log/nmon/incident_$(date +%y%m%d_%H%M%S).nmon
   

3. Post-Incident Analysis: Collect both normal and high-resolution data files for comparison

Containerized Deployment Configuration

When deploying nmon as a DaemonSet, you can configure different behaviors for production vs. troubleshooting:

Production DaemonSet Configuration:

# Default command for production monitoring (30-second intervals)
ENTRYPOINT ["nmon", "-f", "-t", "-s", "30", "-c", "2880"]

Troubleshooting DaemonSet Configuration:

Create a separate DaemonSet or modify the existing one for incident response:

# In your DaemonSet YAML, you can override the command for troubleshooting
containers:
- name: nmon
  image: <your-registry>/nmon-repo:v1
  command: ["nmon", "-f", "-t", "-p", "-T", "-s", "1", "-c", "1800"]
  # ... rest of configuration

Or use environment variables for flexible configuration:

containers:
- name: nmon
  image: <your-registry>/nmon-repo:v1
  env:
  - name: NMON_INTERVAL
    value: "1"  # Change to 30 for production
  - name: NMON_COUNT
    value: "1800"  # Adjust based on monitoring duration needed
  - name: NMON_FLAGS
    value: "-t -p -T"  # Include top processes, process details, and threads
  command: ["/bin/sh", "-c", "nmon -f $NMON_FLAGS -s $NMON_INTERVAL -c $NMON_COUNT"]

Real-World Example: Capturing Process Information During CPU Spikes

Based on production experience with OKE clusters experiencing intermittent CPU spikes:

Problem: CPU spikes lasting less than a minute causing hundreds of alerts, but standard monitoring couldn’t identify the root cause.

Solution: Deploy nmon with proper configuration:

# Deploy nmon DaemonSet with 30-second intervals for continuous monitoring
kubectl apply -f nmon-daemonset-production.yaml

# When CPU spike is detected, immediately capture high-resolution data
kubectl exec -n monitoring <nmon-pod-name> -- nmon -f -t -p -s 1 -c 300 -F /opt/nmon/output/spike_$(date +%y%m%d_%H%M%S).nmon

Key Learning: The -t flag is essential. Without it, you’ll capture system metrics but won’t be able to identify which processes are consuming CPU during the spike.

Configuration Best Practices

1. Always use -t flag: This is non-negotiable for troubleshooting. It enables process-level visibility.

2. Match interval to problem duration:
   • For issues lasting seconds: Use 1-second intervals
   • For issues lasting minutes: 5-10 second intervals may suffice
   • For continuous monitoring: 30-60 second intervals

3. Balance data volume with detail: Higher resolution generates more data. Plan storage accordingly.

4. Use -p and -T selectively: These flags add overhead. Use them during active troubleshooting, not for continuous monitoring.

5. File naming convention: Use descriptive names with timestamps for easy identification:

   nmon -f -t -s 1 -c 600 -F /var/log/nmon/cpu_spike_$(date +%y%m%d_%H%M%S).nmon
   

Analyzing nmon Data

nmon generates files with the .nmon extension, which contain all the collected metrics in a CSV-like format. There are several ways to analyze this data:

Option 1: Using nmonchart (Recommended for Visual Analysis)

The simplest and most effective approach is to use the included nmonchart tool to generate interactive HTML-based charts:

./nmonchart/nmonchart hostname_date_time.nmon
# Creates: hostname_date_time.html

This generates an interactive HTML file with comprehensive performance visualizations. The following sections show examples of the key charts available:

CPU Utilization Chart

The CPU utilization chart shows how CPU resources are distributed across different categories:

Figure 1: CPU Utilization Chart showing User%, System%, Wait%, Idle%, and Steal% over time. High Wait% (orange area) indicates I/O bottlenecks. This example shows data collected at 10-second intervals - use 1-second intervals for troubleshooting incidents.

Memory Usage Chart

The memory chart displays RAM utilization breakdown:

Figure 2: Real Memory (RAM) breakdown showing total, free, cached, active, buffers, and inactive memory over time. Helps identify memory pressure that might correlate with performance issues.

Disk I/O Charts

Disk charts help identify I/O bottlenecks:

Figure 3: Disk Busy Percentage (stacked) showing activity across multiple disk devices. High values indicate disk I/O bottlenecks.

Figure 4: Top 15 disks by Busy% - identifies which specific disks are experiencing the highest utilization.

Run Queue Chart

The run queue chart shows process scheduling activity:

Figure 5: Run Queue showing Runnable (blue) and Blocked (red) processes. High runnable counts indicate CPU contention where many processes are competing for CPU time.

Key Features of nmonchart Visualizations:

• Time-series Analysis: All charts are time-aligned, making it easy to correlate events across different metrics
• Interactive Navigation: Scroll through the HTML to view different time periods and metrics
• Configuration Summary: The HTML includes system configuration details (CPU count, memory, disk layout) captured at collection time

Option 2: Custom Analysis with nmon2csv

For more advanced analysis, you can convert nmon data to CSV format and process it with tools like Python, R, or Excel:

# Install nmon2csv (if not already available)
pip install nmon2csv

# Convert nmon data to CSV
nmon2csv hostname_date_time.nmon

Option 3: Integration with Monitoring Platforms

For persistent monitoring, you can integrate nmon data with platforms like Prometheus and Grafana:

1. Use a sidecar container to process nmon output
2. Expose metrics in Prometheus format
3. Create Grafana dashboards for visualization

Use Case: Identifying CPU-Intensive Processes During Intermittent Spikes

Let’s walk through a real-world use case: identifying CPU spikes in an OKE cluster that last less than a minute.

Scenario Setup

Initial State: nmon DaemonSet running with 30-second intervals for continuous production monitoring.

Incident: Intermittent CPU spikes detected, causing hundreds of alerts but standard monitoring cannot identify root cause.

Investigation Steps

1. Deploy nmon for Production Monitoring (if not already running)

   # Deploy with 30-second intervals for continuous monitoring
   kubectl apply -f nmon-daemonset.yaml
   

2. When CPU Spike is Detected: Immediately switch to high-resolution capture

   # Execute high-resolution capture in the running nmon pod
   kubectl exec -n monitoring <nmon-pod-name> -- nmon -f -t -p -s 1 -c 300 -F /opt/nmon/output/spike_$(date +%y%m%d_%H%M%S).nmon
   

   Critical: Ensure the -t flag is included to capture top processes. Without this flag, you’ll only see system-level metrics and won’t be able to identify the problematic processes.

3. Collect nmon Files after the incident

   # Copy nmon files from the pod
   kubectl cp monitoring/<nmon-pod-name>:/opt/nmon/output/spike_*.nmon ./incident_data.nmon
   

4. Generate Visualizations using nmonchart

   ./nmonchart/nmonchart incident_data.nmon
   # Opens incident_data.html in your browser
   

5. Analyze Visualizations: Review the HTML charts to identify the issue
   • CPU Chart (Figure 1): Locate the spike time window and check Wait% for I/O bottlenecks
   • Run Queue Chart (Figure 5): Check for high runnable processes indicating CPU contention
   • Top Processes Section (Figure 6): Identify which processes consumed CPU during the spike - requires -t flag
   • Memory Chart (Figure 2): Check if memory pressure coincided with CPU spike
   • Disk I/O Charts (Figures 3-4): Determine if disk I/O was a contributing factor
   • Process Correlation (Figure 7): Review bubble chart for multi-dimensional resource analysis
6. Extract Process Details from raw nmon data (if needed for further analysis)

   # Extract top processes during the spike period
   grep "^TOP" incident_data.nmon | tail -50
      
   # Look for processes with high CPU usage
   grep "^TOP" incident_data.nmon | grep -E "CPU.*[5-9][0-9]|CPU.*100"
   

7. Identify Root Cause: Correlate findings from visualizations (Figures 1-7) with application logs and system events

Key Success Factors

• The -t flag is essential: Without it, process identification is impossible
• High-resolution timing: 1-second intervals capture short-lived spikes that 30-second intervals miss
• Immediate response: Start high-resolution capture as soon as the issue is detected
• Process details: The -p flag provides command arguments that help identify specific application components

Example Output Analysis

When analyzing nmon data, look for patterns like:

TOP,CPU%,Mem%,Pid,Command
TOP,45.2,2.1,12345,/usr/bin/java -Xmx2g MyApplication
TOP,32.1,1.5,12346,/usr/bin/kubelet --config=/etc/kubernetes/kubelet.conf

This shows that a Java application process (PID 12345) is consuming 45.2% CPU during the spike, which would be missed with standard monitoring intervals.

When visualized using nmonchart (see Figure 6), you can see this process activity clearly in the Top Process Commands chart. The 1-second granularity allows you to catch short-lived spikes that might be missed with standard monitoring intervals, while the -t flag ensures you can identify exactly which processes are responsible.

Correlating Charts for Root Cause Analysis:

When analyzing performance incidents, correlate findings across multiple charts:

• High CPU Wait% (Figure 1) + High Disk Busy% (Figures 3-4) = I/O bottleneck
• High Runnable processes (Figure 5) + High CPU User% (Figure 1) = CPU contention
• Specific process CPU spikes (Figure 6) + Memory pressure (Figure 2) = Process-specific resource issue
• Process correlation (Figure 7) helps identify processes with high resource usage across multiple dimensions

Best Practices for nmon in OKE

1. Configuration Strategy:
   • Use 30-second intervals for continuous production monitoring to balance detail with overhead
   • Switch to 1-second intervals only when troubleshooting active incidents
   • Always include the -t flag to capture top processes (critical for troubleshooting)
2. Resource considerations:
   • While nmon is lightweight, 1-second intervals generate significantly more data than 30-second intervals
   • Monitor the resource usage of nmon itself, especially during high-resolution captures
   • Plan storage capacity based on your monitoring intervals and retention requirements
3. Data retention:
   • Implement a rotation policy for nmon files to prevent disk space issues
   • Archive high-resolution troubleshooting captures separately from routine monitoring data
   • Consider compressing older nmon files to save space
4. Security context:
   • The nmon container requires privileged access to gather all metrics
   • Consider your security requirements and use least-privilege principles where possible
   • Review and restrict access to nmon output files containing system information
5. Selective deployment:
   • You may not need nmon on every node
   • Consider using node selectors to deploy only to nodes of interest
   • For troubleshooting, you can deploy to specific nodes experiencing issues
6. Process visibility:
   • Always use -t flag: This is the most critical flag for troubleshooting. Without it, you cannot identify which processes are consuming resources
   • Use -p flag during active troubleshooting to get command arguments for better process identification
   • Use -T flag when investigating thread-level issues
7. Integration with alerting:
   • Automate the analysis of nmon data to trigger alerts on abnormal patterns
   • Set up automated high-resolution capture when critical alerts are triggered
   • Correlate nmon data with application and infrastructure logs for comprehensive analysis
8. Troubleshooting Workflow:
   • Establish baseline metrics with normal interval monitoring
   • When issues occur, immediately switch to high-resolution capture
   • Document the configuration used during each troubleshooting session
   • Review and refine your monitoring strategy based on incident patterns

Conclusion

nmon provides a powerful and flexible way to capture performance data in OKE clusters, helping identify and troubleshoot issues that might be missed with standard monitoring solutions. By deploying nmon as a DaemonSet with appropriate configuration for your use case, you gain detailed visibility into process behavior across your cluster.

Key Takeaways:

• Use appropriate intervals: 30-second intervals for production monitoring, 1-second intervals for active troubleshooting
• Always include the -t flag: This is critical for capturing top processes and identifying resource-consuming applications
• Dynamic configuration: Switch to high-resolution mode when investigating incidents, then return to normal monitoring
• Process identification: The combination of -t, -p, and -T flags provides comprehensive process and thread-level visibility

This approach complements existing monitoring solutions, providing the flexibility to capture high-resolution data when needed while maintaining efficient continuous monitoring during normal operations. The ability to dynamically adjust monitoring resolution based on operational needs makes nmon an invaluable tool for troubleshooting complex performance issues in OKE environments.

References

• Monitor Linux performance with nmon
• Monitoring Linux with nmon
• nmon Documentation
• nmonchart Documentation
• Oracle Kubernetes Engine Documentation

Description: After setting up Fusion Applications audit logs ingestion in OCI Log Analytics (Parts 1 and 2), organizations need to transform raw log data into actionable insights. This blog post demonstrates how to leverage OCI Log Analytics dashboards and LoganAI to gain comprehensive visibility into Fusion Applications ESS jobs, security events, and audit activities. We’ll explore dashboard visualizations, compare Fusion native interfaces with OCI Log Analytics capabilities, and showcase AI-powered analysis for proactive monitoring and threat detection.

Introduction

Welcome to Part 3 of our blog series, where we move beyond simply collecting Fusion Applications audit logs and start putting them to use for practical, real-world monitoring of your organization’s Fusion security and operational health.

Fusion Audit Log Key Use Cases Overview:

Figure 1: Key Fusion Audit Logs Use Cases for OCI Log Analytics

In Parts 1 and 2, you learned how to configure permissions and set up OCI Log Analytics to ingest audit logs from Oracle Fusion Applications. Now that your data is flowing in, the main challenge is turning raw log data into insights that improve your security posture and operations—especially for Fusion ESS (Enterprise Scheduler Service) and Security and Audit use cases.

While Oracle Fusion offers basic native tools for tracking scheduled jobs and audit events, those interfaces are often limited: you might find it hard to review historical trends, correlate security activities across modules, or customize dashboards for stakeholders. Most importantly, as your monitoring needs evolve, you will need deeper analytics and a more proactive approach.

Figure 2: Fusion Applications Native Scheduled Processes Work Area - Typical user interface for ESS job tracking

Historically, to answer critical questions like below, organizations have developed a set of dashboards and reports, set up rule-based alerts, and sifted through log records to identify patterns.

• What patterns do you see in off-hours logins?
• Which countries and cities have the highest number of failed login attempts?
• Which users have logged in from geographically distant locations in the past 24 hours?
• Show me users with unusual login patterns that might indicate compromised accounts
• Identify any brute-force attack patterns in failed login attempts
• Are there trends in failed logins or system access?
• Are there patterns in job delays or recurring errors across modules?
• What ESS jobs are failing most frequently, and why?
• Show me ESS jobs that are taking longer than usual to execute compared to last month
• Identify any unusual patterns in ESS job execution that might indicate performance degradation
• Which ESS jobs are at risk of failure based on recent trends?
• What is causing ESS job performance degradation in the CRM module?

At Oracle, we have collective knowledge domain expertise in Fusion Applications and OCI Log Analytics, and we have developed a set of dashboards and reports that provide a comprehensive view of Fusion Applications audit logs and security events. We have published these dashboards as out-of-the-box in the Log Analytics Dashboards section, or you can find more technical content in the Oracle o11y solutions GitHub repository.

• Oracle Fusion Apps_ESS Analysis: Provides a comprehensive view of Enterprise Scheduler Service (ESS) job execution status, including total jobs run, success and failure rates, historical trends, and quick identification of failed or delayed jobs.
• Oracle Fusion Apps_User Access Analysis: Offers in-depth insight into Fusion user access events, including login/logout activity, unique user counts, client locations, failed login attempts, and suspicious sign-in/out patterns across modules.
• Oracle Fusion Apps_OPSS Audit Analysis: Focuses on OPSS (Oracle Platform Security Services) audit events, surfacing privileged account use, policy changes, and security-related configuration activity for compliance and incident investigation.
• Oracle Fusion Apps_SOA: Tracks key SOA (Service-Oriented Architecture) process activity, identifying integration errors, process execution status, and bottlenecks in connected systems.
• Oracle Fusion Apps_Data Service (MDS) Unified: Unifies visibility into Fusion Metadata Services (MDS) activities, surfacing configuration changes and important metadata management events across modules.
• Oracle Human Capital Management_Business Objects: Monitors critical transactions and access patterns related to HCM business objects, including changes, workflows, and sensitive personnel data activity.
• Oracle Fusion Apps_OIC Process Monitoring: Monitors Oracle Integration Cloud (OIC) process executions, integration flows, and failures relevant to Fusion Apps, providing operational assurance for cloud-based integrations.

Understanding the Oracle Fusion Apps Enterprise Scheduler Service (ESS) Analysis Dashboard

The ESS Analysis Dashboard is a comprehensive view of ESS job execution status that provides proactive monitoring and rapid troubleshooting capabilities. It aggregates ESS audit log data ingested via the Management Agent REST API source (configured in Part 2) and presents it through multiple visualization widgets. The following screenshot shows the ESS Analysis Dashboard:

Figure 3: ESS Analysis Dashboard - Comprehensive view of ESS job execution status

Each widget serves a specific monitoring purpose:

• Real-time Job Status Monitoring: Instantly see the overall health of ESS jobs, including total jobs, success rates, and failures, without digging into each job.
• Historical Trend Analysis: View long-term trends and patterns in job performance to spot slowdowns or recurring issues over time.
• Cross-Module Job Correlation: See ESS job data from all Fusion modules together, making it easy to identify problems that span multiple modules.
• Performance Degradation Detection: Quickly find jobs that are running slower than usual so you can fix problems before failures occur.
• Schedule Optimization Insights: Use heatmaps to spot busy periods and scheduling conflicts, so you can better plan and spread out job workloads.

The ESS Analysis Dashboard provides several key advantages for day-to-day operations:

• At-a-Glance Health Monitoring: Administrators can quickly assess overall ESS job health for a period of time without navigating through multiple Fusion screens or manually aggregating data. The dashboard’s summary widgets (counters, donut charts, status distributions) provide immediate visibility into job execution status.

   <copy>
   'Log Source' = 'Oracle Fusion Apps: Enterprise Scheduler Service' | stats distinctcount('Request ID') as Requests by 'State Description'
   </copy>
  

  Figure 4: ESS Job Status Distribution

• Rapid Failure Identification: By clicking the number of failed jobs, the log explorer will open another tab with the failed jobs details and the corresponding query. The query is as follows:

   <copy>
   'Log Source' = 'Oracle Fusion Apps: Enterprise Scheduler Service' and 'State Description' = error | timestats distinctcount('Request ID') as Requests by 'State Description'
   </copy>
  

  Figure 5: Failed Jobs Details

• Schedule Conflict Prevention: The ESS Job Schedule Heatmap (Widget 5) visualizes execution patterns across time periods, helping administrators identify and resolve scheduling conflicts before they impact job execution.

   <copy>
   'Log Source' = 'Oracle Fusion Apps: Enterprise Scheduler Service' | stats distinctcount('Request ID') as Requests by hour('Start Time')
   </copy>
  

  Figure 6: ESS Job Schedule Heatmap

• Performance Trend Awareness: Historical trend analysis helps administrators identify jobs that are gradually slowing down, allowing proactive optimization before failures occur. Widgets like “ESS Job Execution Duration Analysis” (Widget 4) and “Jobs with Execution Time Degradation” (Widget 7) surface these trends automatically.

   <copy>
   'Log Source' = 'Oracle Fusion Apps: Enterprise Scheduler Service' | timestats avg(ElapsedTime) as 'Avg Duration', max(ElapsedTime) as 'Max Duration' by 'Job Display Name'
   </copy>
  

  Figure 7: ESS Job Execution Duration Analysis

• Cross-Module Impact Assessment: When investigating job failures or performance issues, administrators can quickly see which Fusion modules are affected and correlate issues across different application areas. This unified view is essential for understanding the broader impact of ESS job problems.

   <copy>
   'Log Source' = 'Oracle Fusion Apps: Enterprise Scheduler Service' | stats distinctcount('Request ID') as Requests, avg(ElapsedTime) as 'Avg Duration' by 'Product Family'
   </copy>
  

  Figure 8: ESS Job Module Performance Impact

Understanding the Oracle Fusion Apps User Access Analysis Dashboard

The User Access Analysis Dashboard is a comprehensive view of Fusion user access events that provides proactive monitoring and rapid troubleshooting capabilities. It aggregates Fusion user access audit log data ingested via the Management Agent REST API source (configured in Part 2) and presents it through multiple visualization widgets. The following screenshot shows the User Access Analysis Dashboard:

Figure 9: User Access Analysis Dashboard - Comprehensive view of Fusion user access events

Each widget serves a specific monitoring purpose:

• Real-time Login Activity Monitoring: Instantly see the overall health of Fusion user access, including total login events, success and failure rates, without digging into each login event.
• Historical Trend Analysis: View long-term trends and patterns in login activity to spot slowdowns or recurring issues over time.
• Cross-Module Login Correlation: See Fusion user access data from all Fusion modules together, making it easy to identify problems that span multiple modules.
• Geographic Anomaly Detection: Identify unusual geographic access patterns that may indicate account compromise or unauthorized access.

• Failed Login Analysis: Track and analyze failed login attempts to identify potential security threats and brute-force attacks.

• Fusion User Access Login Events Analysis: This widget displays login event counts by status (Success, Failed), providing an overview of authentication activity and helping identify authentication failures that may indicate security threats.

   <copy>
   'Log Source' = 'Fusion Apps: Sign In - Sign Out Activity Logs' and 'User ID' != rats_monitor and 'User ID' != fusion_apps_osn_integ_appid | stats count as 'Login Events ' by Status | sort -Status
   </copy>
  

  Figure 10: Fusion User Access Login Events Analysis

• Fusion User Access Unique Users Analysis: This widget tracks the number of unique users accessing Fusion Applications over time, enabling administrators to monitor user activity trends and detect unusual spikes or drops in user engagement.

   <copy>
   'Log Source' = 'Fusion Apps: Sign In - Sign Out Activity Logs' and 'User ID' != rats_monitor and 'User ID' != fusion_apps_osn_integ_appid | stats distinctcount('User ID') as 'Unique Users', trend(distinctcount('User ID'))
   </copy>
  

  Figure 11: Fusion User Access Unique Users Analysis

• Fusion User Access Geographic Client Locations Analysis: This widget visualizes user access by geographic location on a world map, helping identify impossible travel scenarios and unusual geographic access patterns that may indicate account compromise or unauthorized access.

   <copy>
   'Log Source' = 'Fusion Apps: Sign In - Sign Out Activity Logs' and 'User ID' != rats_monitor and 'User ID' != fusion_apps_osn_integ_appid | stats distinctcount('User ID') as 'Unique Users' by 'Client Host Country', 'Client Host City'
   </copy>
  

  Figure 12: Fusion User Access Geographic Client Locations Analysis

• Fusion Apps: Failed Login User by City: This widget shows failed login attempts grouped by city, helping identify geographic hotspots for authentication failures that may indicate brute-force attacks or compromised credentials.

   <copy>
   'Log Source' = 'Fusion Apps: Sign In - Sign Out Activity Logs' and 'User ID' != rats_monitor and 'User ID' != fusion_apps_osn_integ_appid and 'Status' = 'Failed' | stats count as FailedLoginCount by 'Client Host City'
   </copy>
  

  Figure 13: Fusion Apps: Failed Login User by City

• Fusion Apps: Failed Login Top Users: This widget identifies users with the highest number of failed login attempts, helping prioritize investigation of potentially compromised accounts or brute-force attack targets.

   <copy>
   'Log Source' = 'Fusion Apps: Sign In - Sign Out Activity Logs' and 'User ID' != rats_monitor and 'User ID' != fusion_apps_osn_integ_appid and 'Status' = 'Failed' | stats count as FailedLoginCount by 'User Name'
   </copy>
  

  Figure 14: Fusion Apps: Failed Login Top Users

• Fusion Apps: Failed Login multi-dimensional Analysis: This widget provides a multi-dimensional view of failed logins by country and city, enabling correlation analysis to identify coordinated attacks across geographic regions.

   <copy>
   'Log Source' = 'Fusion Apps: Sign In - Sign Out Activity Logs' and 'User ID' != rats_monitor and 'User ID' != fusion_apps_osn_integ_appid and 'Status' = 'Failed' | stats count as FailedLoginCount by 'Client Host Country', 'Client Host City'
   </copy>
  

  Figure 15: Fusion Apps: Failed Login multi-dimensional Analysis

AI Augmentation: Enriching Fusion Log Analytics with GenAI-Powered Insights

With the introduction of the GenAI-powered LoganAI feature in Log Analytics, this process is dramatically enhanced. Now, administrators and security teams can simply ask their questions in plain language—LoganAI automatically leverages the underlying log data as context for the large language model (LLM). This means you can get clear, actionable answers to complex operational and security queries in real time, without manually writing queries or manually preparing the data.

Key benefits of integrating LoganAI with Fusion log monitoring include:

• Natural Language Insights: Ask LoganAI direct questions like “Show the top failed login users this month” or “Summarize the recent security exceptions,” and receive tailored analytics and context from your logs, instantly.
• Contextual, Explainable Summaries: LoganAI processes large volumes of log data—clustering, detecting patterns, and providing summarized explanations of job executions, user activities, or anomalies, turning raw logs into operational intelligence.
• Proactive Threat Detection: By connecting the dots across audit and application logs, LoganAI can uncover rare risks or unusual activity patterns—empowering you to detect threats before they escalate.
• AI-Augmented Dashboards: Your existing widgets and dashboards are automatically enriched with AI-powered explanations, making log data comprehensible and actionable for users of all backgrounds, not just data experts.

By leveraging LoganAI’s GenAI capabilities, you move from static metrics to dynamic, AI-driven observability—enabling faster, deeper insights into both ESS job processing and Fusion user access, all by simply providing your logs as context to the LLM.

This section showcases how LoganAI elevates Fusion operational and security monitoring with GenAI-powered, context-aware intelligence and a conversational approach, making advanced log analytics accessible and actionable for everyone.

Prerequisite: Enabling LoganAI for AI-Powered Analysis

LoganAI is a generative AI-powered feature in Log Analytics that enables advanced natural language understanding, summarization, in-context assistance, and intelligent insights across various Log Analytics workflows. It leverages the OCI Generative AI service in your tenancy. Enabling LoganAI may incur additional charges depending on usage. Please refer to OCI Generative AI Pricing for more details. As a tenancy administrator, you can enable or disable LoganAI features in specific OCI regions and configure how Generative AI services will be used within Log Analytics.

That’s where OCI Log Analytics and AI-based augmentation (with LoganAI) make a significant difference. With these tools, you can:

• Visualize and monitor ESS job health, failures, and performance in dynamic dashboards.
• Detect security threats as they emerge, leveraging both built-in analytics and customizable real-time alerts.
• Unlock long-term and cross-module visibility for audit logs and security events—far beyond the short history available in native Fusion screens.
• Take advantage of AI-powered features such as natural language queries and anomaly detection to find patterns or issues you might otherwise miss.

Step 1: Enable LoganAI

• Navigate to Log Analytics Administration:
  • Go to OCI Console > Observability & Management > Log Analytics
  • Click Administration in the left navigation menu
  • Click Service Details in the left navigation menu
  • Click Configure LoganAI Settings in Logan AI section
  • 
  • Figure 16: Configure LoganAI Settings
  • Update the **Preference settings for current region **
  • Confirm the setting to Enable LoganAI for the tenancy
  • Confirm the Oracle LLM license details
  • Select License acceptance requirements for users
  • Add the required policies in IAM Policies section

  •  allow group Log-Analytics-Users to use generative-ai-chat in compartment id ocid1.compartment.oc1..aaaaaaaawaaaaaaaaaaazma
     allow group Log-Analytics-Users to read generative-ai-model in compartment id ocid1.compartment.oc1..aaaaaaaawaaaaaaaaaaazma
     allow group Log-Analytics-Users to read generative-ai-endpoint in compartment id ocid1.compartment.oc1..aaaaaaaawaaaaaaaaaaazma
    

  • Click Update LoganAI Settings to save the changes
  • 
  • Figure 17: LoganAI Settings Updated

Step 2: Using LoganAI in Log Explorer

Once enabled, LoganAI is available throughout Log Explorer. The Explain feature uses artificial intelligence to summarize log messages and charts, transforming complex log data into actionable insights. To access LoganAI Explain, click the AI icon wherever it appears throughout Log Explorer. The AI icon is available in various visualizations that present log records. LoganAI Explain can be utilized in the following ways:

LoganAI Explain Use Cases

Single Log Explain: Click the AI icon next to any log record to instantly receive a concise summary of that entry. Ask follow-up questions or review conversation history for deeper insights—ideal for quickly understanding errors or suspicious events.

Multi-log Explain: Summarize and analyze multiple log entries at once by clicking the AI icon above the log records. LoganAI automatically highlights key patterns and anomalies across your selection, helping you spot trends or root causes without manual review.

Cluster Explain: Use Cluster Explain to review large volumes of similar logs. With one click, AI summarizes patterns, outliers, and potential issues within log clusters—supporting rapid identification of systemic problems or anomalies in operational data.

Chart Explain: In Link chart visualizations, click the AI icon below the chart to get an AI-generated summary of key patterns, trends, and anomalies. You can target specific data subsets by selecting chart bubbles for focused analysis.

For more details, visit the Oracle Log Analytics documentation.

Let’s see the LoganAI Explain in action with sample queries:

• LoganAI Cluster Explain Insights: What patterns do you see in off-hours (8am-8pm EST) logins?

  • Query: ‘Log Source’ = ‘Fusion Apps: Sign In - Sign Out Activity Logs’ and ‘User ID’ not like in (rats_monitor, ‘Fusion_app%’) and ‘User ID’ != fusion_apps_osn_integ_appid

    timestats distinctcount(‘User ID’) as logrecords by ‘Log Source’

  • LoganAI explain:
    • Visualization: Records with Histogram
    • Time Range: Last 14 days
    • Screenshot:
• LoganAI Chart Explain Insights: Show me users with unusual login patterns that might indicate compromised accounts

  • Query: ‘Log Source’ = ‘Fusion Apps: Sign In - Sign Out Activity Logs’ and ‘User ID’ != rats_monitor and Status = fail and ‘User ID’ != fusion_apps_osn_integ_appid

    link ‘User ID’

    stats unique(‘Client Host Country’) as Country

    eval ‘Time Window’ = ‘End Time’ - ‘Start Time’

    rename ‘End Time’ as ‘Last failed login’

    rename Count as ‘Failed Logins’

    rename ‘Start Time’ as ‘First failed login’

    classify correlate = -*, Country ‘First failed login’, ‘Failed Logins’, ‘Time Window’, ‘User ID’ as ‘Failed Logins Analysis’

  • LoganAI explain:
    • Visualization: Link
    • Time Range: Last 7 days
    • Screenshot:
• LoganAI Chart Explain Insights: Which countries and cities have the highest number of failed login attempts?

  • Query: ‘Log Source’ = ‘Fusion Apps: Sign In - Sign Out Activity Logs’ and ‘User ID’ != rats_monitor and ‘User ID’ != fusion_apps_osn_integ_appid

    link Status, ‘Client Host Country’

    stats unique(‘Client Host City’) as ‘Client Host City’, unique(‘Client Host Region’) as ‘Client Host Region’

    classify topcount = 300 correlate = -*, ‘Client Host Country’, Status ‘Start Time’, ‘Client Host City’, Count as ‘Start Time, Client Host City, Client Host Country’

  • LoganAI explain:
    • Visualization: Link
    • Time Range: Last 7 days
    • Screenshot:
• LoganAI Single Log Explain Insights: What is the ESS job about?

  • Query: ‘Log Source’ = ‘Oracle Fusion Apps: Enterprise Scheduler Service’

    timestats distinctcount(‘Request ID’) as Jobs

  • LoganAI explain:
    • Visualization: Records with Histogram
    • Time Range: Last 7 days
    • Screenshot:
• LoganAI Multiple Log Explain Insights: Which job has the longest elapsed time?

  • Query: ‘Log Source’ = ‘Oracle Fusion Apps: Enterprise Scheduler Service’

    timestats count as logrecords by ‘Log Source’

  • LoganAI explain:
    • Visualization: Records with Histogram
    • Time Range: Last 8 hours
    • Screenshot:
• LoganAI Chart Explain Insights: What ESS jobs are failing most frequently, and what are the primary reasons for the failure?

  • Query: ‘Log Source’ = ‘Oracle Fusion Apps: Enterprise Scheduler Service’

    link ‘Request ID’

    stats unique(Application) as Application, avg(ElapsedTime) as E, unique(‘Instance Parent Id’) as ‘Parent ID’, unique(‘Job Display Name’) as Job, unique(‘Parent Request ID’) as ‘Parent Req ID’, unique(‘Product Name’) as Product, unique(‘Product Family’) as ‘Product Family’, latest(‘State Description’) as Status, unique(Submitter) as Submitter

    eval ‘Time Taken’ = unit(E, ms)

    eval U = url(‘https://oracle.com’, View)

    eventstats distinctcount(‘Request ID’) as Requests, avg(E) as AE, latest(U) as Details by Product, Job, Status

    eval ‘Average Time Taken’ = unit(AE, ms)

    createtable name = ‘Job Summary’ select Product, ‘Product Family’, Job, Status, Requests, ‘Average Time Taken’, Details

    classify ‘Start Time’, Job, ‘Time Taken’, Product, ‘Product Family’, Status as ‘ESS Jobs Analysis’

    fields -E, -AE

  • LoganAI explain:
    • Visualization: Link
    • Time Range: Last 7 days
    • Screenshot:
• LoganAI Chart Explain Insights: What is causing ESS job performance degradation in the CRM module?

  • Query: ‘Log Source’ = ‘Oracle Fusion Apps: Enterprise Scheduler Service’

    cluster

  • LoganAI explain:
    • Visualization: Cluster
    • Time Range: Last 7 days
    • Screenshot:

Best Practices for Maximizing LoganAI Explain

To fully leverage LoganAI’s advanced AI-driven analysis capabilities within OCI Log Analytics, adopt these best practices:

1. Use ‘Log Source’ or ‘Entity’ Filtering with Cluster Visualization
   Always apply appropriate ‘Log Source’ or ‘Entity’ filters before using Cluster visualization. This ensures that the AI summary is both relevant and focused, and enables context-rich follow-up queries for deeper insight. Precise filtering reduces noise and guides LoganAI to more actionable conclusions.

2. Select the Proper Time Range for the Right Level of Analysis
   For a high-level overview or trend discovery, set broader time ranges when clustering data. For detailed, tactical analysis—such as troubleshooting or anomaly investigation—narrow the time window and use Records with Histogram visualization. Matching your time range to your investigation depth yields optimal insights and avoids information overload.

3. Be Aware of Cluster Group Context Window Limit (500 Items)
   LoganAI Cluster analysis works best with smaller, well-scoped data sets. The maximum context window for a cluster group is 500 records; if your data approaches or exceeds this, the AI may hallucinate or miss important context details. For accurate root cause analysis and precise recommendations, focus on filtering to smaller, meaningful log groups.

4. Leverage Link Visualization and Charts for AI-Augmented Context
   Use Link visualization and custom charts to enrich your analysis. By combining AI-derived insights with visual context, you gain a clearer picture of relationships (e.g., between users, locations, and event statuses) and subtle patterns not easily detected by raw log inspection alone.

5. Utilize the Custom Action Menu to Add and Augment Datasets
   Take advantage of LoganAI in the custom action menu to bring additional datasets into your investigation. This multi-source context augments the analysis, uncovering trends or anomalies that might span modules or log types, and yielding holistic, cross-source insights.

6. Monitor and Manage Generative AI Usage and Cost
   Each LoganAI Explain action consumes resources from OCI Generative AI Services, and costs are based on how often you use Explain, the size and complexity of the data, and the number of follow-up analyses. These charges are separate from standard OCI Log Analytics costs. Review the OCI Pricing guide under Generative AI Services, and plan your usage accordingly to balance insights value and budget considerations.

By following these best practices, you can maximize both the technical and business value of LoganAI in your security, compliance, and operational monitoring workflows—gaining sharper insights, accelerating investigations, and optimizing your OCI resource usage.

Conclusion

Robust log collection, when combined with AI-driven analytics, unlocks transformative security monitoring, observability, and compliance for Oracle Fusion Applications customers. With the right permissions and access in place, organizations can efficiently ingest and analyze log data using Log Analytics and advanced tooling like LoganAI. This enables deep visibility into operations, rapid threat detection, and identification of anomalous behaviors and attack patterns—all from within unified dashboards and automated workflows.

LoganAI is more than a feature—it’s a leap forward in log intelligence. It makes troubleshooting faster, insights clearer, and log analytics more accessible. Whether you’re an engineer, analyst, or business stakeholder, LoganAI transforms how you understand and act on your log data.

References

• Enable LoganAI
• Create Dashboards
• Log Explorer Query Language
• Create Detection Rules
• Create Alerts for Detected Events
• AI-Driven IAM Audit Analysis for Oracle SaaS
• Beyond Monitoring: AI-Driven Fusion ESS Job Health Risk Observatory
• Fusion Applications Observability: Collecting ESS Logs
• Oracle o11y Solutions GitHub Repository
• Methods and Ways to Extract Session Information Using OCI Audit
• Configure Fusion Applications Audit and Security Logs Ingestion - Part 1
• Configure Fusion Applications Audit and Security Logs Ingestion - Part 2

Introduction

Oracle Fusion Applications generate extensive audit logs that capture every significant security and business event across your SaaS environment. These logs include user authentication events, authorization decisions, administrative changes, transaction-level activities, and platform-level operations. For organizations running Fusion Applications across multiple modules (ERP, HCM, SCM, CX, EPM), centralizing these audit logs in OCI Log Analytics enables:

OCI Log Analytics provides the ideal destination for Fusion Applications audit data, offering powerful exploration capabilities, correlation with other log sources, detection rules, and AI-powered analysis through LoganAI. By leveraging the REST API-based log collection method with OCI Management Agents, organizations can establish continuous, automated ingestion of Fusion audit logs without complex middleware or custom integrations.

Figure 1: Fusion Application Audit Logs Key Use Cases

Note: If you want to monitor the ESS jobs performance and status, I have published a blog post on the Fusion Enterprise Scheduler Service (ESS) job request logs collection using OCI Management Agent and REST API log collection, you can find it here.

However, accessing and collecting Fusion Application audit logs requires understanding the complex permission model and configuring the appropriate integration mechanisms. This blog post addresses two critical aspects:

1. Part 1: Understanding Required Permissions: What privileges and roles are needed to access Fusion audit log data
2. Part 2: Setting Up Log Ingestion: Step-by-step configuration for Fusion Apps Audit Logs using OCI Management Agent and REST API log collection

Note: This guide focuses on Fusion Applications transaction and business object audit logs accessed via the Fusion Audit REST API. For IAM Identity Domain audit events (sign-in attempts, MFA challenges, session activity), these flow through OCI Audit and can be ingested via Service Connector - see Methods and Ways to Extract Session Information Using OCI Audit for that complementary approach.

Solution Design

Oracle Log Analytics enables continuous collection of Fusion Application audit logs through REST API-based log sources. The Management Agent acts as the collector, periodically invoking Fusion Applications REST APIs to fetch audit log data and forwarding it to Log Analytics for parsing, indexing, and analysis.

Reference Architecture:

Figure 2: Fusion Application Audit Logs Reference Architecture

Prerequisites

Before proceeding with the implementation, ensure you have:

• An active OCI tenancy with appropriate IAM permissions
• Access to Oracle Fusion Applications with administrative privileges
• An OCI Compute instance (Oracle Linux) where the Management Agent will be installed
• Network connectivity between the Management Agent host and your Fusion Applications instance
• Understanding of your Fusion Applications POD URL and LOGIN URL (refer to Doc ID 2661308.1 in Oracle My Support)
• Create a service account in Fusion Applications for the Management Agent to use

Part 2: Step-by-Step Setup for Ingesting Fusion Application Audit Logs

This section provides detailed step-by-step instructions for setting up Fusion Application audit log collection using the Management Agent and REST API log ingestion method.

Note: OCI Log Analytics provides pre-defined log sources and parsers for Fusion Applications audit logs.

• Fusion Apps: SOA Audit Logs
• Fusion Apps: Sign In - Sign Out Activity Logs
• Fusion Apps: OPSS Audit Logs
• Fusion Apps: ODI Audit Logs
• Fusion Apps: OBIEE Audit Logs
• Fusion Apps: MDS Audit Logs
• Fusion Apps: HCM People Audit Logs
• Fusion Apps: HCM Payroll Audit Logs

Prerequisites

Before you set up log ingestion for Fusion Application Audit logs, ensure the following prerequisites are met:

• Review the Fusion Applications REST API documentation to understand how to use the Audit Logs APIs, available endpoints, supported log types, and query mechanisms.

• Ensure you have valid Fusion Applications credentials (username and password) with sufficient privileges to access the Audit Logs REST API. Privileges are typically assigned via appropriate Fusion Applications roles.

• Identify and note the required endpoints for your Fusion application instance:
  • login_url: The login URL for Fusion Applications (e.g., https://<identity-domain>.fa.<region>.oraclecloud.com)
  • pod_url: The base URL for your Fusion Applications pod (e.g., https://<identity-domain>.fa.<region>.oraclecloud.com)
  • proxy_url (Optional): If your environment requires access through a proxy, provide the proxy server URL (http(s)://<proxy-host>:<proxy-port>).
  • For more information on constructing and locating Fusion Application instance URLs, refer to Oracle My Support Doc ID 2661308.1.

  Note: Sign-in Sign-out REST API is deprecated and no longer available after Fusion Identity Upgrade, starting Oracle Fusion 25B or higher. You will have to use the OCI Audit feature as per KB181606

• Ensure Fusion Applications REST API access is enabled for your environment:
  • Confirm that users or application clients have the necessary REST API security privileges and duty roles to access audit logs endpoints.
  • This includes role assignments such as FND_VIEW_AUDIT_HISTORY_PRIV and other relevant product/duty roles depending on the audit log type.
  • REST API access is typically controlled by user provisioning or within the Fusion Applications Security Console.
  • See Fusion Apps REST API Security Overview for details on configuring user access.
• (Optional) If outbound network traffic from the Management Agent host to Fusion Applications must traverse a proxy, ensure the proxy URL and credentials are available. Verify network routing allows connections from your OCI environment to the Fusion Applications REST endpoints.

Step 1: Install and Configure Management Agent

• Provision OCI Compute Instance:
  • Provision an OCI Compute instance with Oracle Linux image
  • Ensure the instance has HTTP/HTTPS access to your Fusion Applications POD URL
  • Note the instance name (e.g., mgmtagent-vm)
• Enable Management Agent Plugin:
  • Navigate to OCI Console > Compute > Instances
  • Select your compute instance
  • Go to the Oracle Cloud Agent tab
  • Enable the Management Agent plugin
  • Ensure the Logging Analytics plugin is deployed during installation
  • Reference: Set Up Continuous Log Collection From Your Hosts
• Fusion Applications Entity Creation:
  • An entity of type “Host (Linux)” will be automatically created in Log Analytics
  • Navigate to OCI Console > Observability & Management > Log Analytics > Administration > Entities
  • Verify the entity exists in the same compartment as your compute instance
  • Note the entity name (should match your instance name, e.g., mgmtagent-vm)
  • Create a new entity with the following details:
    • Name: fusion-apps-mgmtagent-vm
    • Entity Type: Oracle Fusion Applications
    • Management Agent Compartment: your_mgmtagent_compartment_name
    • Management Agent: Agent(mgmtagent-vm)
    • Properties:
      • Property Name: login_url
      • Property Value: The base URL of your Fusion Applications instance (e.g., https://<yourpod>.login.fa.us1.oraclecloud.com)
      • Property Name: pod_url
      • Property Value: The base URL of your Fusion Applications instance (e.g., https://<yourpod>.fa.us1.oraclecloud.com)
      • Property Name: proxy_url
      • Property Value: (Optional) The URL that sends a request to your proxy server
    • Agent Collection Properties:
      • Source Type: REST API
      • Historical Data: 30D (or your desired historical data fetching period)
      • Enable Filter Duplicate Records: true
      • Customized Schedule: FREQ=MINUTELY;INTERVAL=5 (5 mins or your desired schedule interval)
    • Click Create
    • 
    • Figure 3: Fusion Applications Entity Creation
• Configure IAM Policies:
  • Ensure policies are in place to allow Management Agent to send logs
  • Reference: Allow Continuous Log Collection Using Management Agents

Step 2: Configure Management Agent Credentials

• SSH to the Compute Instance:

   ssh opc@<your-mgmtagent-vm-compute-instance-ip>
  

• Switch to Root User (if needed):

   sudo su -
  

• Navigate to Management Agent Bin Directory:
  • For Management Agents running via Oracle Cloud Agent plugin:

    cd /var/lib/oracle-cloud-agent/plugins/oci-managementagent/polaris/agent_inst/bin
    

  • For manually installed Management Agents:

    cd /opt/oracle/mgmt_agent/agent_inst/bin
    

• Create Credential JSON File: Navigate to the appropriate /bin directory for your setup to create the credential JSON file. The following example shows the values provided in the fapps.json file:

   cat > /tmp/FA-CREDS.json << EOF
   {
         "source": "lacollector.la_rest_api",
         "name": "FA-CREDS",
         "type": "HTTPSBasicAuthCreds",
         "description": "These are HTTPS (BasicAuth) credentials.",
         "properties": [
            {
               "name": "HTTPSUserName",
               "value": "USER"
            },
            {
               "name": "HTTPSPassword",
               "value": "PASS"
            }
         ]
   }
   EOF
  

  Replace your_service_account_username and your_service_account_password with your Fusion Applications service account credentials.

• Insert Credentials into Agent Credential Store:

   cat fapps.json | ./credential_mgmt.sh -s logan -o upsertCredentials
  

• Verify Credential Storage:

   ./credential_mgmt.sh -o listCredentials -s logan | grep FA-CREDS
  

  Reference: Management Agent Source Credentials

Step 3: Associate Source with Entity and Start Collection

• Validate the Log Source Endpoint Configuration:
  • Navigate to OCI Console > Observability & Management > Log Analytics > Administration > Sources
  • Click on the log source (e.g., Fusion Apps: Sign In - Sign Out Activity Logs)
  • Click Customize
  • Click the endpoint … > View to inspect the endpoint configuration
• Navigate to Entity Details:
  • Go to OCI Console > Observability & Management > Log Analytics > Administration > Entities
  • Click on your entity (e.g., fusion-apps-mgmtagent-vm)
• Add Data Source:
  • Click Add Data
  • Select the entity (e.g., fusion-apps-mgmtagent-vm)
  • Click Next
• Select Log Sources:
  • Search for Fusion Apps: to filter Fusion Applications sources
  • Select the log sources you want to collect (e.g., Fusion Apps: Sign In - Sign Out Activity Logs, Fusion Apps: OPSS Audit Logs, Fusion Apps: ESS Audit Logs)
  • 
  • Figure 4: Log Analytics Fusion Apps Audit Select Log Sources
  • Verify the Agent Collection Properties are correctly configured
  • Select the correct log group to store log data for new association
  • Click Validate and configure log collection
  • 
  • Figure 5: Log Analytics Fusion Apps Audit Validate and Configure Log Collection
• Start Collection:
  • Once configured, click Take me to Log Explorer to verify log ingestion

Step 4: Verify Log Ingestion

• Navigate to Log Explorer:
  • Go to OCI Console > Observability & Management > Log Analytics > Log Explorer
  • Or use the link provided after configuration
• Query Audit Logs:
  • In the search bar, enter a query to find Fusion Applications Sign-in/Sign-out activity logs:

    'Log Source' in ('Fusion Apps: ESS Audit Logs', 'Oracle Fusion Apps: OPSS Audit Logs', 'Fusion Apps: OPSS Audit Logs', 'Fusion Apps: MDS Audit Logs') | stats count as logrecords by 'Log Source' | sort -logrecords
    

  • To verify sign-in/sign-out activity (if enabled):

    'Log Source' = 'Fusion Apps: Sign In - Sign Out Activity Logs' | stats count as logrecords by 'Log Source' | sort -logrecords
    

  • Adjust the time range to see recent logs
  • Click Run Query

Step 5: Import Dashboards

• Locate Dashboards on GitHub:
  • Find the pre-built Log Analytics dashboard JSON files for Fusion Apps audit logs on the Oracle o11y solutions GitHub repository.
  • Common dashboard files:
    • Oracle Fusion User Access.json
    • Oracle Fusion Apps_OPSS Audit Analysis.json
    • Oracle Fusion Apps_SOA.json
    • Oracle Fusion Apps_Data Service (MDS) Unified.json
    • Oracle Human Capital Management_Business Objects.json
    • Fusion OIC Process Monitoring.json
• Import the Dashboard in OCI Log Analytics:
  • Go to OCI Console → Observability & Management → Log Analytics → Dashboards.
  • Click on Import dashboards.
  • Browse to the downloaded dashboard .json file and select it.

Conclusion

By following the process outlined in this guide, you can establish a robust and streamlined approach to collecting, monitoring, and analyzing audit logs from Oracle Fusion Applications within OCI Log Analytics. Automating audit log ingestion via REST APIs and the OCI Management Agent empowers your organization to consolidate critical audit and user activity data in one place—enabling you to:

• Increase Visibility and Control: Monitor user interactions and system events across Fusion Applications to strengthen governance and transparency.
• Simplify Compliance Reporting: Generate comprehensive audit trails that support both external regulatory requirements and internal policies.
• Drive Operational Insight: Leverage centralized log data to uncover usage trends, improve business processes, and support data-driven decision making.
• Accelerate Threat Detection: Quickly identify anomalies and act on potential threats using advanced analytics and visualization capabilities.

Integrating your Fusion Applications audit logs with OCI Log Analytics not only simplifies adherence to audit and compliance obligations, but also transforms raw log data into actionable intelligence. This approach enhances your organization’s ability to safeguard data, comply with regulations, and maximize the value derived from your Oracle Cloud investments.

References

• ESS Roles Documentation
• Enable Audit for Oracle Platform Security Services
• Set Up REST API Log Collection
• Ingest Fusion Applications Audit Logs
• Fusion Applications Audit Policies
• Fusion Applications Audit Report API
• HCM REST API Authorization
• How to Get User Sign-in Sign-out Information?
• Create a Fusion Service Integration User Account
• REST API Quick Start
• Set Up Continuous Log Collection From Your Hosts
• Allow Continuous Log Collection Using Management Agents
• Management Agent Source Credentials
• Doc ID 2661308.1 - Fusion Applications URLs and Configuration
• Doc ID 2059102.1 - How To Configure And Use Audit In BI Publisher For Fusion SaaS Cloud Customers
• Doc ID 2731495.1 - Fusion Applications OTBI: How To Configure And Use Audit For Usage and Performance In OTBI For Fusion SaaS Cloud Customers

Introduction

Oracle Fusion Applications generate extensive audit logs that capture every significant security and business event across your SaaS environment. These logs include user authentication events, authorization decisions, administrative changes, transaction-level activities, and platform-level operations. For organizations running Fusion Applications across multiple modules (ERP, HCM, SCM, CX, EPM), centralizing these audit logs in OCI Log Analytics enables:

OCI Log Analytics provides the ideal destination for Fusion Applications audit data, offering powerful exploration capabilities, correlation with other log sources, detection rules, and AI-powered analysis through LoganAI. By leveraging the REST API-based log collection method with OCI Management Agents, organizations can establish continuous, automated ingestion of Fusion audit logs without complex middleware or custom integrations.

Figure 1: Fusion Application Audit Logs Key Use Cases

Note: If you want to monitor the ESS jobs performance and status, I have published a blog post on the Fusion Enterprise Scheduler Service (ESS) job request logs collection using OCI Management Agent and REST API log collection, you can find it here.

However, accessing and collecting Fusion Application audit logs requires understanding the complex permission model and configuring the appropriate integration mechanisms. This blog post addresses two critical aspects:

1. Part 1: Understanding Required Permissions: What privileges and roles are needed to access Fusion audit log data
2. Part 2: Setting Up Log Ingestion: Step-by-step configuration for Fusion Apps Audit Logs using OCI Management Agent and REST API log collection

Note: This guide focuses on Fusion Applications transaction and business object audit logs accessed via the Fusion Audit REST API. For IAM Identity Domain audit events (sign-in attempts, MFA challenges, session activity), these flow through OCI Audit and can be ingested via Service Connector - see Methods and Ways to Extract Session Information Using OCI Audit for that complementary approach.

Solution Design

Oracle Log Analytics enables continuous collection of Fusion Application audit logs through REST API-based log sources. The Management Agent acts as the collector, periodically invoking Fusion Applications REST APIs to fetch audit log data and forwarding it to Log Analytics for parsing, indexing, and analysis.

Reference Architecture:

Figure 2: Fusion Application Audit Logs Reference Architecture

Prerequisites

Before proceeding with the implementation, ensure you have:

• An active OCI tenancy with appropriate IAM permissions
• Access to Oracle Fusion Applications with administrative privileges
• An OCI Compute instance (Oracle Linux) where the Management Agent will be installed
• Network connectivity between the Management Agent host and your Fusion Applications instance
• Understanding of your Fusion Applications POD URL and LOGIN URL (refer to Doc ID 2661308.1 in Oracle My Support)
• Create a service account in Fusion Applications for the Management Agent to use

Part 1: Understanding Required Fusion Applications Audit and Security Logs Permissions

Accessing Fusion Application audit logs requires a combination of Fusion Applications privileges. This section provides a comprehensive overview of the permission requirements.

Service Account Setup

For REST API access, you need to create a service account (integration user) in Fusion Applications:

• Sign in to Oracle Fusion Applications using administrator privileges.
• Navigate to:
  • Tools > Security Console > Users
• Create a service account (integration user) with the following details:
  • Last Name: SERVICE_APP_ICS_ID
  • Email: Valid email address
  • User Name: SERVICE_APP_ICS_ID
  • Person Type: None

Recommended Core Audit and Security Logs Privileges

Sign In - Sign Out Activity Logs

You can get the last seven days of user sign-in sign-out information using a setting available on the Add User Account page in Security Console. To view the setting, you must enable a profile option.

Note: Update the configuration of the Sign-In/Sign-Out audit data requires Fusion Apps Administrator or Security Administrator privilege, or make sure the admin account user has FND_MANAGE_AUDIT_POLICIES_PRIV.

• The Sign-In/Sign-Out audit data requires an additional Fusion Applications enablement step that is separate from role/privilege assignment.
• Data availability limitation: Sign-In/Sign-Out audit data is available for the last seven days only.
• Enable the profile option:
  • In Setup and Maintenance
  • Open the side bar to search task
  • 
  • Figure 3: Fusion Applications Audit LogsSearch task
  • 
  • Figure 4: Fusion Applications Audit Logs Search task Manage Administrator Profile Values
  • Open the task Manage Administrator Profile Values
  • Search for Profile Option Code: ASE_ADVANCED_USER_MANAGEMENT_SETTING
  • Set the profile option value Site to Yes
  • Click Save and Close
  • 
  • Figure 5: Fusion Applications Audit Logs Manage Administrator Profile Values
• Enable administration access for the integration user:
  • In Security Console, go to Users
  • Open the integration user (or when creating it via Add User Account)
  • In Advanced Information, enable Enable Administration Access for Sign In-Sign Out Audit REST API
• Reference: How to Get User Sign-in Sign-out Information?

Fusion ERP, HCM, SCM, CX Audit Logs

• Sign in to Oracle Fusion Applications using administrator privileges.
• Navigate to:
  • Tools > Security Console > Roles
• Create a service account custom role with the following details:
  • Role Name: SERVICE_APP_ICS_ID_ROLE
  • Role Code: SERVICE_APP_ICS_ID_ROLE
  • Role Category: Common - Job Roles
  • Role Privilege:
    • Add the following Function Security Policy ASE_REST_SERVICE_ACCESS_IDENTITY_INTEGRATION_PRIV
    • 
    • Figure 6: Fusion Applications Audit Logs Add Function Security Policy
    • Click Next
  • Add User SERVICE_APP_ICS_ID to the custom role SERVICE_APP_ICS_ID_ROLE
  • Click Save and Close

Oracle Platform Security Services (OPSS) Audit Logs

Oracle Platform Security Services (OPSS) audit enables auditing of security-related events, such as authentication and authorization, across Oracle Fusion Applications. To enable OPSS audit logging, follow these steps:

By default, auditing is disabled for all Oracle Fusion Applications, including Oracle Platform Security Services (OPSS). To enable or disable auditing for OPSS, follow these steps:

• Access Manage Audit Policies:
  • In the Setup and Maintenance work area, search for and select the Manage Audit Policies task.
  • 
  • Figure 7: Fusion Applications Audit Logs Search task Manage Audit Policies
• Set Audit Level for OPSS:
  • On the Manage Audit Policies page, locate Oracle Platform Security Services in the list.
  • To enable auditing, set the Audit Level to Low - Critical Events Only.
  • To disable auditing, set the Audit Level to None.
  • Click Save and Close to confirm and apply your changes.

Note:

• To manage audit policies, you must have the Manage Audit Policies (FND_MANAGE_AUDIT_POLICIES_PRIV) function security privilege. The predefined Application Implementation Consultant job role includes this privilege. If you want to want to use the Security - Audit Real Time subject area in Oracle Transactional Business Intelligence (OTBI) to access OPSS audit data. Access to this OTBI subject area requires the FBI_SECURITY_TRANSACTION_ANALYSIS_DUTY duty role. The predefined IT Security Manager job role contains this duty role.

Enterprise Scheduler Service (ESS) Job Request Logs

Users can view and manage the scheduled processes that they submit, for example, to cancel the process or see the output. However, to perform these actions for scheduled processes submitted by other users, specific roles are required.

ESS Monitor Role is sufficient for viewing the scheduled process history.

The privilege required to access for all Scheduled Process is ESS_REQUEST_HISTORY.

• Open Security Console and go to the Roles page. Click Create Role.
• (Optional) On the Data Security Policies page:
  • Add a policy with:
    • Data Resource: ESS_REQUEST_HISTORY
    • Data Set: All Values
    • Actions: Select desired tasks (e.g., View, Update, Cancel).
• On the Role Hierarchy page, add any needed roles.
• On the Users page, assign the users who need this custom role.

Enterprise Scheduler Service (ESS) Audit Logs

You can also update the Audit Level for Oracle Enterprise Scheduling Service (ESS) for ESS metadata changes, ESS request operations, Administrative operations to ESS request processor and dispatcher.

• Access Manage Audit Policies:
  • In the Setup and Maintenance work area, search for and select the Manage Audit Policies task.
• Set Audit Level for ESS:
  • On the Manage Audit Policies page, locate Enterprise Scheduler Service in the list.
  • To enable auditing, set the Audit Level to Medium - Some Events.
  • Click Save and Close to confirm and apply your changes.

Oracle Metadata Services (MDS) Audit Logs

Customization of the Fusion web application developed with Oracle ADF is supported by Oracle Metadata Services (MDS), which is a framework that integrates with JDeveloper tools and also manages the customization of ADF features by end users at runtime. The Oracle Metadata Services (MDS) framework allows you to create applications that your customers can further customize for their users or customers and which the end users can also customize.

MDS can be applied to the range of other components in the ADF technology stack, such as ADF Faces, ADF Controller, ADF Model, and ADF Business Components. Oracle Metadata Services (MDS) audit enables auditing of metadata changes in Oracle Fusion Applications. To enable MDS audit logging, follow these steps:

• Access Manage Audit Policies:
  • In the Setup and Maintenance work area, search for and select the Manage Audit Policies task.
• Set Audit Level for MDS:
  • On the Manage Audit Policies page, locate Oracle Metadata Services in the list.
  • To enable auditing, set the Audit Level to Medium - Some Events.
  • Click Save and Close to confirm and apply your changes.

Oracle SOA Suite Audit Logs

The Oracle SOA Suite audit enables auditing of SOA Suite events in Oracle Fusion Applications. To enable SOA Suite audit logging, follow these steps:

• Access Manage Audit Policies:
  • In the Setup and Maintenance work area, search for and select the Manage Audit Policies task.
• Set Audit Level for SOA Suite:
  • On the Manage Audit Policies page, locate Oracle SOA Suite in the list.
  • To enable auditing, set the Audit Level to Medium - Some Events.
  • Click Save and Close to confirm and apply your changes.

Oracle Fusion Applications Business Object Audit Logs

To configure which attributes appear in the audit report for Oracle Fusion Applications business objects, follow these steps (adapted from Oracle documentation):

• Access Manage Audit Policies:
  • In the Setup and Maintenance work area, search for and select the Manage Audit Policies task.

• Under Fusion Applications Business Objects section, set Audit Level to Auditing

• Under Fusion Applications Business Objects section, click Configure Business Object Attributes:

• Select Product and Object:
  • From the Product list, select the relevant product. For example:
    • Global Human Resources for Log Analytics Fusion Apps: HCM People Audit Logs Log Source
    • Global Payroll for Log Analytics Fusion Apps: HCM Payroll Audit Logs Log Source
  • Check the Audit checkbox for the relevant business object.
  • 
  • Figure 8: Fusion Applications Audit Logs Configure Business Object Audit HCM People
  • Click Save and Close to confirm and apply your changes.

Conclusion

In Part 1, we explored how to enable and configure audit logging across key Oracle Fusion Applications components, including ESS, MDS, SOA Suite, and Business Objects. By following the steps outlined, you ensure critical business transactions and workflow activities are recorded, supporting your compliance and security monitoring objectives. Properly configured audit policies allow for effective tracking of user and system activity, forming the foundation for robust audit log collection and analysis in OCI Log Analytics.

In the next part, we will cover step-by-step guidance on ingesting these audit logs into Oracle Cloud Infrastructure, so you can gain actionable insights and streamline your audit compliance processes.

References

• ESS Roles Documentation
• Enable Audit for Oracle Platform Security Services
• Fusion Applications Audit Policies
• Fusion Applications Audit Report API
• HCM REST API Authorization
• How to Get User Sign-in Sign-out Information?
• Create a Fusion Service Integration User Account
• REST API Quick Start
• Doc ID 2661308.1 - Fusion Applications URLs and Configuration
• Doc ID 2059102.1 - How To Configure And Use Audit In BI Publisher For Fusion SaaS Cloud Customers
• Doc ID 2731495.1 - Fusion Applications OTBI: How To Configure And Use Audit For Usage and Performance In OTBI For Fusion SaaS Cloud Customers