Gardener

Appearance

Edit this page Report an issue

Falco Configuration ​

Introduction ​

Falco can be enabled and configured in the extensions section of the shoot manifest. The minimal configuration is:

yaml
  - type: shoot-falco-service

Alternatively, you can specify a provider configuration:

yaml
  - type: shoot-falco-service
    providerConfig:
      apiVersion: falco.extensions.gardener.cloud/v1alpha1
      kind: FalcoServiceConfig

This configuration deploys Falco into the kube-system namespace of the shoot cluster. Defaults for Falco versions, Falco rules, and event storage are applied according to the landscape configuration. For most deployments, the defaults are:

  • The latest supported Falco version
  • The Falco rules ruleset, possibly with extensions to avoid false positives in an empty cluster
  • The logging destination

The configuration above will be expanded to:

yaml
    - type: shoot-falco-service
      providerConfig:
        kind: FalcoServiceConfig
        apiVersion: falco.extensions.gardener.cloud/v1alpha1
        falcoVersion: 0.40.0
        autoUpdate: true
        rules:
          standard:
          - falco-rules
        destinations:
        - name: logging

Configuration Details ​

Below is the full configuration, explained in detail:

yaml
  - type: shoot-falco-service
    providerConfig:
      apiVersion: falco.extensions.gardener.cloud/v1alpha1
      kind: FalcoServiceConfig
      # Optional: uses the latest version tagged as "supported" if omitted
      falcoVersion: 0.42.0
      # Optional: defaults to true
      autoUpdate: true|false
      # Optional
      nodeSelector:
        key: value
      falcoConfig:
        # optional resources to be set on Falco pods
        resources:
          limits:
            cpu: 200m
            memory: 1Gi
          requests:
            cpu: 150m
            memory: 512Mi
      tolerations:
      - key: key1
        effect: NoSchedule
        operator: Equal
        value: value1
      # Optional: defaults to false
      heartbeatEvent: true|false
      rules:
        # Standard rules from https://github.com/falcosecurity/rules/tree/main/rules
        standard:
        - falco-rules
        - falco-incubating-rules
        - falco-sandbox-rules
        # Custom rules
        custom:
        - resourceName: rules1
      destinations:
        # Possible values: stdout, logging, custom, opensearch, splunk
        - name: custom
          # Options, may be required to configure destination
          resourceSecretName: secret

Versions and Update Strategy ​

The falcoVersion field specifies the Falco version from the non-expired versions configured in the Falco profile. If omitted, the latest supported version is selected.

Setting autoUpdate to false allows users to opt out of automated Falco updates, for example, if certain rules are incompatible with newer Falco versions. When autoUpdate is true, the extension always updates Falco to the latest supported version in the Falco profile. No automated updates occur if autoUpdate is false, except when the configured version has expired (expirationDate is in the past). In that case, Falco is updated to the next non-expired version.

Configuring Rules ​

The rules section configures both standard and custom Falco rules:

yaml
      rules:
        standard:
        - falco-rules
        - falco-incubating-rules
        - falco-sandbox-rules
        # Custom rules
        custom:
        - resourceName: my-custom-rules
        - resourceName: more-custom-rules
        ...

Falco rules (falco-rules), incubating rules (falco-incubating-rules), and sandbox rules (falco-sandbox-rules) are provided by the Falco community. The Falco rules have been extended to avoid false positives in an empty Kubernetes cluster managed by Gardener. Note that incubating and sandbox rules may still emit false positives. Only the listed values are accepted; rule files do not depend on each other.

The standard rules can be extended or replaced by custom rules. Custom rules in the Gardener setup can be stored in a ConfigMap and applied to the same Gardener project as the cluster shoot manifest. The ConfigMap must be configured as a resource in the cluster as described here.

The my-custom-rules name above requires an entry with the same name in the resources section of the shoot manifest:

yaml
  resources:
  - name: my-custom-rules
    resourceRef:
      apiVersion: v1
      kind: ConfigMap
      name: my-custom-rules-configmap

This entry refers to a ConfigMap named my-custom-rules-configmap:

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-custom-rules-configmap
  namespace: garden-myproject
data:
  myrules.yaml: |-
    # Custom rules go here
    ...
  myrules-extended.yaml: |-
    # More custom rules
    ...

Each key (e.g., myrules.yaml or myrules-extended.yaml) results in one rules file. You can create a ConfigMap like the above with:

bash
kubectl create configmap my-custom-rules-configmap --from-file=myrules.yaml \
            --from-file=myrules-extended.yaml

If ConfigMap size is restricted, you can add gzipped rule files. File names must end with .gz; other compression formats are not supported.

bash
gzip myrules.yaml
gzip myrules-extended.yaml
kubectl create configmap my-custom-rules-configmap --from-file=myrules.yaml.gz \
            --from-file=myrules-extended.yaml.gz

Note: The maximum size of an unzipped rule file is 1MB, and there is a limit of 10 rule files per ConfigMap.

Ordering is important, as Falco rules may extend other rules that must be defined first. The order is:

  1. Falco rules (if specified)
  2. Falco incubating rules (if specified)
  3. Falco sandbox rules (if specified)
  4. Files from the first custom rule ConfigMap specified in the customRules array (ordered lexicographically by filename)
  5. Files from the second custom rule ConfigMap ...

If no rules key is specified, the falco-rules standard rules are set by default; otherwise, no defaults are applied.

Configuring Destinations ​

Falco can post events to several internal and external storage providers:

  • stdout: Write events to the pod log only
  • central: Post events to a central storage, if offered by the infrastructure provider
  • logging: Post events to the local cluster logging stack
  • opensearch: Post events to an OpenSearch cluster for centralized analysis
  • splunk: Post events to a Splunk instance via the HTTP Event Collector (HEC)
  • custom: Post events to a custom web server

Additionally, operators may inject global default destinations (e.g., central-splunk) into your configuration automatically. These appear in the destinations list alongside standard destinations.

More details for each destination are described below.

Each destination supports an optional enabled field:

yaml
      destinations:
      - name: logging
      - name: central-splunk
        enabled: false

Setting enabled: false keeps the destination in the configuration but disables event forwarding to it. If omitted, the destination is enabled by default. This is useful for disabling operator-injected global default destinations on a per-shoot basis.

The custom destination requires additional configuration:

yaml
      destinations:
      - name: custom
        resourceSecretName: custom-secret

The resourceSecretName references a secret defined in the resources section of the shoot manifest:

yaml
  resources:
  - name: custom-secret
    resourceRef:
      apiVersion: v1
      kind: Secret
      name: my-custom-secret-config

The secret contains the following values:

yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-webhook-secret-config
  namespace: <project namespace in garden cluster>
stringData:
  address: "" # URL
  method: POST
  checkcerts: true | false
  customHeaders: |
    Authorization: Bearer <token>
    header1: value1
    ...

Other Options ​

FalcoServiceConfig supports additional options:

yaml
      # Optional
      nodeSelector:
        key: value
        key2: value2
      # Optional
      - key: key1
        effect: NoSchedule
        operator: Equal
        value: value1
      # Optional: defaults to false
      heartbeatEvent: true|false

The nodeSelector option specifies node selectors for Falco pods, following the Kubernetes API specification. Only nodes matching the specified labels will run Falco pods.

The tolerations option allows Falco pods to be scheduled on nodes with taints, such as master/control-plane nodes or nodes marked with the NoSchedule effect. This follows the Kubernetes API specification for tolerations. This is particularly useful when the Falco pods need to run on nodes that have taints like NoSchedule to ensure comprehensive security monitoring.

The heartbeatEvent option enables or disables the heartbeat event, which is a periodic event sent by Falco to indicate it is running and healthy. This is useful for monitoring.

The falcoConfig option allows to configure the resources section that is configured for the Falco container:

      falcoConfig:
        # optional resources to be set on Falco pods
        resources:
          limits:
            cpu: 200m
            memory: 1Gi
          requests:
            cpu: 150m
            memory: 512Mi

Destinations ​

This section provides details about event destinations, use cases, and implementation.

Do Not Forward Falco Events (stdout Option) ​

Events are logged to Falco pod stdout and not forwarded. Falcosidekick is not deployed if stdout is the only event destination. Users can deploy custom tools to scrape pod logs and store events.

Note: Writing logs to stdout forwards events to the cluster Vali database, as logs from pods in the kube-system namespace are generally stored there. Event details may not be stored in an optimal way for analysis. Use the logging destination if you plan to analyze events.

Store Events in the Cluster Logging Stack (logging Option) ​

Falco events are forwarded to the cluster Vali logging database using the Falcosidekick Loki output.

Events can be queried in the Vali section of the cluster Plutono UI using:

logql
{rule=~".+", tags=~".+", source=~".+"}

More details on possible queries can be found in the LogQL documentation.

Note: Events may be retained for a short period and could be overwritten if the Vali database experiences disk pressure. Replicating events to another location may be necessary.

Custom Destination (custom Option) ​

This option forwards Falco events to user-configurable destinations, such as a SIEM system. It requires a destination configuration in a secret referenced by resourceSecretName. The secret's contents configure the Falcosidekick webhook output.

OpenSearch Destination (opensearch Option) ​

This option forwards Falco events to an OpenSearch cluster for centralized security event analysis and long-term storage

Configuration ​

The opensearch destination requires additional configuration:

yaml
      destinations:
      - name: opensearch
        resourceSecretName: opensearch-config

The resourceSecretName references a secret defined in the resources section of the shoot manifest:

yaml
  resources:
  - name: opensearch-config
    resourceRef:
      apiVersion: v1
      kind: Secret
      name: my-opensearch-config

The secret contains the following values:

yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-opensearch-config
  namespace: <project namespace in garden cluster>
stringData:
  # Required: OpenSearch endpoint URL
  hostport: "https://opensearch.example.com:9200"

  # Optional: Index name (default: "falco")
  index: "falco"

  # Optional: Index suffix for rotation (default: "daily")
  # Options: "none", "daily", "monthly", "annually"
  suffix: "daily"

  # Optional: OpenSearch username for authentication
  username: "falco-user"

  # Optional: OpenSearch password for authentication
  password: "secure-password"

  # Optional: Verify SSL certificates (default: false)
  checkcert: "false"

  # Optional: Minimum priority level (default: "debug")
  # Options: "emergency", "alert", "critical", "error", "warning", "notice", "informational", "debug"
  minimumpriority: "debug"

  # Optional: Number of index shards (default: OpenSearch default)
  numberofshards: "1"

  # Optional: Number of index replicas (default: OpenSearch default)
  numberofreplicas: "1"

  # Optional: Flatten nested fields for easier querying (default: false)
  flattenfields: "false"

  # Optional: Custom HTTP headers
  customheaders: |
    X-Custom-Header: value

Features ​

  • Index Rotation: Automatic daily, monthly, or annual index rotation for easier data management
  • Authentication: Support for username/password authentication
  • TLS/SSL: Configurable certificate verification for secure connections
  • Priority Filtering: Filter events by minimum severity level
  • Custom Fields: Enrich events with cluster metadata (cluster_id automatically added)
  • Index Configuration: Configure shards and replicas for optimal performance

Implementation Details ​

Falcosidekick uses the Elasticsearch output for OpenSearch integration, as OpenSearch maintains API compatibility with Elasticsearch. Events are sent using the OpenSearch Bulk API for efficient ingestion.

Index naming pattern: {index}-{suffix} (e.g., falco-2024.12.15 for daily rotation)

Example: Complete Configuration ​

yaml
apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
  name: my-cluster
  namespace: garden-my-project
spec:
  extensions:
    - type: shoot-falco-service
      providerConfig:
        apiVersion: falco.extensions.gardener.cloud/v1alpha1
        kind: FalcoServiceConfig
        destinations:
          - name: opensearch
            resourceSecretName: opensearch-config
          - name: stdout
  resources:
    - name: opensearch-config
      resourceRef:
        apiVersion: v1
        kind: Secret
        name: my-opensearch-config

Querying Events in OpenSearch ​

Once configured, Falco events can be queried in OpenSearch Dashboards:

  1. Create an index pattern: falco-*
  2. Use the Discover tab to explore events
  3. Example query to find high-priority events:
    json
    {
  "query": {
    "bool": {
      "must": [
        { "term": { "priority": "Critical" } }
      ]
    }
  }
}

Monitoring ​

Check Falcosidekick logs to verify OpenSearch connectivity:

bash
kubectl logs -n kube-system -l app.kubernetes.io/name=falcosidekick -f

Verify events are being indexed in OpenSearch:

bash
curl -u username:password "https://opensearch.example.com:9200/falco-*/_count"

Store Events Centrally (central Option) ​

Note: Central storage is optional and may not be provided as part of the Gardener landscape installation.

Falco events are forwarded to central storage via the Falcosidekick webhook output.

The Falco event ingestor provides a REST API to receive Falco events, validates event integrity, stores events in an SQL database, and implements configurable rate limiting per cluster to prevent overload.

The Falco event provider offers a REST API to access the database. Cluster users must present a valid token (with Viewer access for the Gardener project namespace) to retrieve Falco events for their cluster.

Splunk Destination (splunk Option) ​

This option forwards Falco events to a Splunk instance via the HTTP Event Collector (HEC). It uses the Falcosidekick Splunk output.

Configuration ​

The splunk destination requires additional configuration:

yaml
      destinations:
      - name: splunk
        resourceSecretName: splunk-config

The resourceSecretName references a secret defined in the resources section of the shoot manifest:

yaml
  resources:
  - name: splunk-config
    resourceRef:
      apiVersion: v1
      kind: Secret
      name: my-splunk-config

The secret contains the following values:

yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-splunk-config
  namespace: <project namespace in garden cluster>
stringData:
  # Required: Splunk HEC endpoint URL
  host: "https://splunk-hec.example.com/services/collector/event"

  # Required: Splunk HEC token
  token: "your-hec-token"

  # Optional: Verify SSL certificates (default: true)
  checkcert: "true"

  # Optional: Minimum priority level
  # Options: "emergency", "alert", "critical", "error", "warning", "notice", "informational", "debug"
  minimumpriority: "notice"

  # Optional: Custom HTTP headers (YAML map format)
  customheaders: |
    X-Custom-Header: value

Prerequisites ​

Before configuring the Splunk destination, ensure that:

  1. The Splunk HEC is enabled on your Splunk instance
  2. A HEC token has been created with appropriate permissions
  3. The HEC endpoint is reachable from the shoot cluster (the host field must include the full URL path)

Known Issues ​

Falco 0.41.x - Container Name Display Issue ​

Issue: In Falco versions 0.41.0 through 0.41.3, there is a known bug where the container.name field in alerts displays the container.id instead of the actual container name. See GitHub Issue #3631 for details.

Resolution: This issue is fixed in Falco 0.42.0. If you need accurate container names in alerts, use Falco 0.40.0 or upgrade to 0.42.0.

Affected Versions: 0.41.0, 0.41.3 Fixed In: 0.42.0