-
Notifications
You must be signed in to change notification settings - Fork 897
/
Copy pathdata-model.md
1297 lines (1137 loc) · 41.7 KB
/
data-model.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
# Logs Data Model
**Status**: [Stable](../document-status.md)
<details>
<summary>Table of Contents</summary>
<!-- toc -->
- [Design Notes](#design-notes)
* [Requirements](#requirements)
* [Definitions Used in this Document](#definitions-used-in-this-document)
+ [Type `any`](#type-any)
+ [Type `map`](#type-mapstring-any)
* [Field Kinds](#field-kinds)
- [Log and Event Record Definition](#log-and-event-record-definition)
* [Field: `Timestamp`](#field-timestamp)
* [Field: `ObservedTimestamp`](#field-observedtimestamp)
* [Trace Context Fields](#trace-context-fields)
+ [Field: `TraceId`](#field-traceid)
+ [Field: `SpanId`](#field-spanid)
+ [Field: `TraceFlags`](#field-traceflags)
* [Severity Fields](#severity-fields)
+ [Field: `SeverityText`](#field-severitytext)
+ [Field: `SeverityNumber`](#field-severitynumber)
+ [Mapping of `SeverityNumber`](#mapping-of-severitynumber)
+ [Reverse Mapping](#reverse-mapping)
+ [Error Semantics](#error-semantics)
+ [Displaying Severity](#displaying-severity)
+ [Comparing Severity](#comparing-severity)
* [Field: `Body`](#field-body)
* [Field: `Resource`](#field-resource)
* [Field: `InstrumentationScope`](#field-instrumentationscope)
* [Field: `Attributes`](#field-attributes)
+ [Errors and Exceptions](#errors-and-exceptions)
- [Example Log Records](#example-log-records)
- [Appendix A. Example Mappings](#appendix-a-example-mappings)
* [RFC5424 Syslog](#rfc5424-syslog)
* [Windows Event Log](#windows-event-log)
* [SignalFx Events](#signalfx-events)
* [Splunk HEC](#splunk-hec)
* [Log4j](#log4j)
* [Zap](#zap)
* [Apache HTTP Server access log](#apache-http-server-access-log)
* [CloudTrail Log Event](#cloudtrail-log-event)
* [Google Cloud Logging](#google-cloud-logging)
- [Elastic Common Schema](#elastic-common-schema)
- [Appendix B: `SeverityNumber` example mappings](#appendix-b-severitynumber-example-mappings)
- [References](#references)
<!-- tocstop -->
</details>
This is a data model and semantic conventions that allow to represent logs from
various sources: application log files, machine generated events, system logs,
etc. Existing log formats can be unambiguously mapped to this data model.
Reverse mapping from this data model is also possible to the extent that the
target log format has equivalent capabilities.
The purpose of the data model is to have a common understanding of what a log
record is, what data needs to be recorded, transferred, stored and interpreted
by a logging system.
This proposal defines a data model for
[Standalone Logs](../glossary.md#standalone-log).
## Design Notes
### Requirements
The Data Model was designed to satisfy the following requirements:
- It should be possible to unambiguously map existing log formats to this Data
Model. Translating log data from an arbitrary log format to this Data Model
and back should ideally result in identical data.
- Mappings of other log formats to this Data Model should be semantically
meaningful. The Data Model must preserve the semantics of particular elements
of existing log formats.
- Translating log data from an arbitrary log format A to this Data Model and
then translating from the Data Model to another log format B ideally must
result in a meaningful translation of log data that is no worse than a
reasonable direct translation from log format A to log format B.
- It should be possible to efficiently represent the Data Model in concrete
implementations that require the data to be stored or transmitted. We
primarily care about 2 aspects of efficiency: CPU usage for
serialization/deserialization and space requirements in serialized form. This
is an indirect requirement that is affected by the specific representation of
the Data Model rather than the Data Model itself, but is still useful to keep
in mind.
The Data Model aims to successfully represent 3 sorts of logs and events:
- System Formats. These are logs and events generated by the operating system
and over which we have no control - we cannot change the format or affect what
information is included (unless the data is generated by an application which
we can modify). An example of system format is Syslog.
- Third-party Applications. These are generated by third-party applications. We
may have certain control over what information is included, e.g. customize the
format. An example is Apache log file.
- First-party Applications. These are applications that we develop and we have
some control over how the logs and events are generated and what information
we include in the logs. We can likely modify the source code of the
application if needed.
### Definitions Used in this Document
In this document we refer to types `any` and `map<string, any>`, defined as
follows.
#### Type `any`
Value of type `any` can be one of the following:
- A scalar value: number, string or boolean,
- A byte array,
- An array (a list) of `any` values,
- A `map<string, any>`.
#### Type `map<string, any>`
Value of type `map<string, any>` is a map of string keys to `any` values. The
keys in the map are unique (duplicate keys are not allowed). The representation
of the map is language-dependent.
Arbitrary deep nesting of values for arrays and maps is allowed (essentially
allows to represent an equivalent of a JSON object).
### Field Kinds
This Data Model defines a logical model for a log record (irrespective of the
physical format and encoding of the record). Each record contains 2 kinds of
fields:
- Named top-level fields of specific type and meaning.
- Fields stored as `map<string, any>`, which can contain arbitrary values of
different types. The keys and values for well-known fields follow semantic
conventions for key names and possible values that allow all parties that work
with the field to have the same interpretation of the data. See references to
semantic conventions for `Resource` and `Attributes` fields and examples in
[Appendix A](#appendix-a-example-mappings).
The reasons for having these 2 kinds of fields are:
- Ability to efficiently represent named top-level fields, which are almost
always present (e.g. when using encodings like Protocol Buffers where fields
are enumerated but not named on the wire).
- Ability to enforce types of named fields, which is very useful for compiled
languages with type checks.
- Flexibility to represent less frequent data as `map<string, any>`. This
includes well-known data that has standardized semantics as well as arbitrary
custom data that the application may want to include in the logs.
When designing this data model we followed the following reasoning to make a
decision about when to use a top-level named field:
- The field needs to be either mandatory for all records or be frequently
present in well-known log and event formats (such as `Timestamp`) or is
expected to be often present in log records in upcoming logging systems (such
as `TraceId`).
- The field’s semantics must be the same for all known log and event formats and
can be mapped directly and unambiguously to this data model.
Both of the above conditions were required to give the field a place in the
top-level structure of the record.
## Log and Event Record Definition
[Appendix A](#appendix-a-example-mappings) contains many examples that show how
existing log formats map to the fields defined below. If there are questions
about the meaning of the field reviewing the examples may be helpful.
Here is the list of fields in a log record:
Field Name |Description
---------------|--------------------------------------------
Timestamp |Time when the event occurred.
ObservedTimestamp|Time when the event was observed.
TraceId |Request trace id.
SpanId |Request span id.
TraceFlags |W3C trace flag.
SeverityText |The severity text (also known as log level).
SeverityNumber |Numerical value of the severity.
Body |The body of the log record.
Resource |Describes the source of the log.
InstrumentationScope|Describes the scope that emitted the log.
Attributes |Additional information about the event.
Below is the detailed description of each field.
### Field: `Timestamp`
Type: Timestamp, uint64 nanoseconds since Unix epoch.
Description: Time when the event occurred measured by the origin clock, i.e. the
time at the source. This field is optional, it may be missing if the source
timestamp is unknown.
### Field: `ObservedTimestamp`
Type: Timestamp, uint64 nanoseconds since Unix epoch.
Description: Time when the event was observed by the collection system. For
events that originate in OpenTelemetry (e.g. using OpenTelemetry Logging SDK)
this timestamp is typically set at the generation time and is equal to
Timestamp. For events originating externally and collected by OpenTelemetry
(e.g. using Collector) this is the time when OpenTelemetry's code observed the
event measured by the clock of the OpenTelemetry code. This field SHOULD be set
once the event is observed by OpenTelemetry.
For converting OpenTelemetry log data to formats that support only one timestamp
or when receiving OpenTelemetry log data by recipients that support only one
timestamp internally the following logic is recommended:
- Use `Timestamp` if it is present, otherwise use `ObservedTimestamp`.
### Trace Context Fields
#### Field: `TraceId`
Type: byte sequence.
Description: Request trace id as defined in
[W3C Trace Context](https://www.w3.org/TR/trace-context/#trace-id). Can be set
for logs that are part of request processing and have an assigned trace id. This
field is optional.
#### Field: `SpanId`
Type: byte sequence.
Description: Span id. Can be set for logs that are part of a particular
processing span. If SpanId is present TraceId SHOULD be also present. This field
is optional.
#### Field: `TraceFlags`
Type: byte.
Description: Trace flag as defined in
[W3C Trace Context](https://www.w3.org/TR/trace-context/#trace-flags)
specification. At the time of writing the specification defines one flag - the
SAMPLED flag. This field is optional.
### Severity Fields
#### Field: `SeverityText`
Type: string.
Description: severity text (also known as log level). This is the original
string representation of the severity as it is known at the source. If this
field is missing and `SeverityNumber` is present then the short name that
corresponds to the `SeverityNumber` may be used as a substitution. This field is
optional.
#### Field: `SeverityNumber`
Type: number.
Description: numerical value of the severity, normalized to values described in
this document. This field is optional.
`SeverityNumber` is an integer number. Smaller numerical values correspond to
less severe events (such as debug events), larger numerical values correspond to
more severe events (such as errors and critical events). The following table
defines the meaning of `SeverityNumber` value:
SeverityNumber range|Range name|Meaning
--------------------|----------|-------
1-4 |TRACE |A fine-grained debugging event. Typically disabled in default configurations.
5-8 |DEBUG |A debugging event.
9-12 |INFO |An informational event. Indicates that an event happened.
13-16 |WARN |A warning event. Not an error but is likely more important than an informational event.
17-20 |ERROR |An error event. Something went wrong.
21-24 |FATAL |A fatal error such as application or system crash.
Smaller numerical values in each range represent less important (less severe)
events. Larger numerical values in each range represent more important (more
severe) events. For example `SeverityNumber=17` describes an error that is less
critical than an error with `SeverityNumber=20`.
#### Mapping of `SeverityNumber`
Mappings from existing logging systems and formats (or **source format** for
short) must define how severity (or log level) of that particular format
corresponds to `SeverityNumber` of this data model based on the meaning given
for each range in the above table.
If the source format has more than one severity that matches a single range in
this table then the severities of the source format must be assigned numerical
values from that range according to how severe (important) the source severity
is.
For example if the source format defines "Error" and "Critical" as error events
and "Critical" is a more important and more severe situation then we can choose
the following `SeverityNumber` values for the mapping: "Error"->17,
"Critical"->18.
If the source format has only a single severity that matches the meaning of the
range then it is recommended to assign that severity the smallest value of the
range.
For example if the source format has an "Informational" log level and no other
log levels with similar meaning then it is recommended to use
`SeverityNumber=9` for "Informational".
Source formats that do not define a concept of severity or log level MAY omit
`SeverityNumber` and `SeverityText` fields. Backend and UI may represent log
records with missing severity information distinctly or may interpret log
records with missing `SeverityNumber` and `SeverityText` fields as if the
`SeverityNumber` was set equal to INFO (numeric value of 9).
#### Reverse Mapping
When performing a reverse mapping from `SeverityNumber` to a specific format
and the `SeverityNumber` has no corresponding mapping entry for that format
then it is recommended to choose the target severity that is in the same
severity range and is closest numerically.
For example Zap has only one severity in the INFO range, called "Info". When
doing reverse mapping all `SeverityNumber` values in INFO range (numeric 9-12)
will be mapped to Zap’s "Info" level.
#### Error Semantics
If `SeverityNumber` is present and has a value of ERROR (numeric 17) or higher
then it is an indication that the log record represents an erroneous situation.
It is up to the reader of this value to make a decision on how to use this fact
(e.g. UIs may display such errors in a different color or have a feature to find
all erroneous log records).
If the log record represents an erroneous event and the source format does not
define a severity or log level concept then it is recommended to set
`SeverityNumber` to ERROR (numeric 17) during the mapping process. If the log
record represents a non-erroneous event the `SeverityNumber` field may be
omitted or may be set to any numeric value less than ERROR (numeric 17). The
recommended value in this case is INFO (numeric 9). See
[Appendix B](#appendix-b-severitynumber-example-mappings) for more mapping
examples.
#### Displaying Severity
The following table defines the recommended short name for each
`SeverityNumber` value. The short name can be used for example for representing
the `SeverityNumber` in the UI:
SeverityNumber|Short Name
--------------|----------
1 |TRACE
2 |TRACE2
3 |TRACE3
4 |TRACE4
5 |DEBUG
6 |DEBUG2
7 |DEBUG3
8 |DEBUG4
9 |INFO
10 |INFO2
11 |INFO3
12 |INFO4
13 |WARN
14 |WARN2
15 |WARN3
16 |WARN4
17 |ERROR
18 |ERROR2
19 |ERROR3
20 |ERROR4
21 |FATAL
22 |FATAL2
23 |FATAL3
24 |FATAL4
When an individual log record is displayed it is recommended to show both
`SeverityText` and `SeverityNumber` values. A recommended combined string in
this case begins with the short name followed by `SeverityText` in parenthesis.
For example "Informational" Syslog record will be displayed as **INFO
(Informational)**. When for a particular log record the `SeverityNumber` is
defined but the `SeverityText` is missing it is recommended to only show the
short name, e.g. **INFO**.
When drop down lists (or other UI elements that are intended to represent the
possible set of values) are used for representing the severity it is preferable
to display the short name in such UI elements.
For example a dropdown list of severities that allows filtering log records by
severities is likely to be more usable if it contains the short names of
`SeverityNumber` (and thus has a limited upper bound of elements) compared to a
dropdown list, which lists all distinct `SeverityText` values that are known to
the system (which can be a large number of elements, often differing only in
capitalization or abbreviated, e.g. "Info" vs "Information").
#### Comparing Severity
In the contexts where severity participates in less-than / greater-than
comparisons `SeverityNumber` field should be used. `SeverityNumber` can be
compared to another `SeverityNumber` or to numbers in the 1..24 range (or to the
corresponding short names).
### Field: `Body`
Type: any.
Description: A value containing the body of the log record (see the description
of `any` type above). Can be for example a human-readable string message
(including multi-line) describing the event in a free form or it can be a
structured data composed of arrays and maps of other values. First-party
Applications SHOULD use a string message. However, a structured body SHOULD be
used to preserve the semantics of structured logs emitted by Third-party
Applications. Can vary for each occurrence of the event coming from the same
source. This field is optional.
### Field: `Resource`
Type: `map<string, any>`.
Description: Describes the source of the log, aka
[resource](../overview.md#resources). Multiple occurrences of events coming from
the same event source can happen across time and they all have the same value of
`Resource`. Can contain for example information about the application that emits
the record or about the infrastructure where the application runs. Data formats
that represent this data model may be designed in a manner that allows the
`Resource` field to be recorded only once per batch of log records that come
from the same source. SHOULD follow OpenTelemetry
[semantic conventions for Resources](../resource/semantic_conventions/README.md).
This field is optional.
### Field: `InstrumentationScope`
Type: (Name,Version) tuple of strings.
Description: the [instrumentation scope](../glossary.md#instrumentation-scope).
Multiple occurrences of events coming from the same scope can happen across time and
they all have the same value of `InstrumentationScope`. For log sources which define
a logger name (e.g. Java
[Logger Name](https://docs.oracle.com/javase/7/docs/api/java/util/logging/Logger.html#getLogger(java.lang.String)))
the Logger Name SHOULD be recorded as the Instrumentation Scope name.
Version is optional. Name SHOULD be specified if version is specified, otherwise Name
is optional.
### Field: `Attributes`
Type: `map<string, any>`.
Description: Additional information about the specific event occurrence. Unlike
the `Resource` field, which is fixed for a particular source, `Attributes` can
vary for each occurrence of the event coming from the same source. Can contain
information about the request context (other than TraceId/SpanId). SHOULD follow
OpenTelemetry [semantic conventions for Log Attributes](./semantic_conventions/README.md) or
[semantic conventions for Span Attributes](../trace/semantic_conventions/README.md).
This field is optional.
#### Errors and Exceptions
Additional information about errors and/or exceptions that are associated with
a log record MAY be included in the structured data in the `Attributes` section
of the record.
If included, they MUST follow the OpenTelemetry
[semantic conventions for exception-related attributes](./semantic_conventions/exceptions.md#attributes).
## Example Log Records
For example log records see
[JSON File serialization](../protocol/file-exporter.md#examples).
## Appendix A. Example Mappings
This section contains examples of mapping of other events and logs formats to
this data model.
### RFC5424 Syslog
<table>
<tr>
<td>Property</td>
<td>Type</td>
<td>Description</td>
<td>Maps to Unified Model Field</td>
</tr>
<tr>
<td>TIMESTAMP</td>
<td>Timestamp</td>
<td>Time when an event occurred measured by the origin clock.</td>
<td>Timestamp</td>
</tr>
<tr>
<td>SEVERITY</td>
<td>enum</td>
<td>Defines the importance of the event. Example: `Debug`</td>
<td>Severity</td>
</tr>
<tr>
<td>FACILITY</td>
<td>enum</td>
<td>Describes where the event originated. A predefined list of Unix processes. Part of event source identity. Example: `mail system`</td>
<td>Attributes["syslog.facility"]</td>
</tr>
<tr>
<td>VERSION</td>
<td>number</td>
<td>Meta: protocol version, orthogonal to the event.</td>
<td>Attributes["syslog.version"]</td>
</tr>
<tr>
<td>HOSTNAME</td>
<td>string</td>
<td>Describes the location where the event originated. Possible values are FQDN, IP address, etc.</td>
<td>Resource["host.hostname"]</td>
</tr>
<tr>
<td>APP-NAME</td>
<td>string</td>
<td>User-defined app name. Part of event source identity.</td>
<td>Resource["service.name"]</td>
</tr>
<tr>
<td>PROCID</td>
<td>string</td>
<td>Not well defined. May be used as a meta field for protocol operation purposes or may be part of event source identity.</td>
<td>Attributes["syslog.procid"]</td>
</tr>
<tr>
<td>MSGID</td>
<td>string</td>
<td>Defines the type of the event. Part of event source identity. Example: "TCPIN"</td>
<td>Attributes["syslog.msgid"]</td>
</tr>
<tr>
<td>STRUCTURED-DATA</td>
<td>array of maps of string to string</td>
<td>A variety of use cases depending on the SDID:
Can describe event source identity
Can include data that describes particular occurrence of the event.
Can be meta-information, e.g. quality of timestamp value.</td>
<td>SDID origin.swVersion map to Resource["service.version"]
SDID origin.ip map to attribute[net.host.ip"]
Rest of SDIDs -> Attributes["syslog.*"]</td>
</tr>
<tr>
<td>MSG</td>
<td>string</td>
<td>Free-form text message about the event. Typically human readable.</td>
<td>Body</td>
</tr>
</table>
### Windows Event Log
<table>
<tr>
<td>Property</td>
<td>Type</td>
<td>Description</td>
<td>Maps to Unified Model Field</td>
</tr>
<tr>
<td>TimeCreated</td>
<td>Timestamp</td>
<td>The time stamp that identifies when the event was logged.</td>
<td>Timestamp</td>
</tr>
<tr>
<td>Level</td>
<td>enum</td>
<td>Contains the severity level of the event.</td>
<td>Severity</td>
</tr>
<tr>
<td>Computer</td>
<td>string</td>
<td>The name of the computer on which the event occurred.</td>
<td>Resource["host.hostname"]</td>
</tr>
<tr>
<td>EventID</td>
<td>uint</td>
<td>The identifier that the provider used to identify the event.</td>
<td>Attributes["winlog.event_id"]</td>
</tr>
<tr>
<td>Message</td>
<td>string</td>
<td>The message string.</td>
<td>Body</td>
</tr>
<tr>
<td>Rest of the fields.</td>
<td>any</td>
<td>All other fields in the event.</td>
<td>Attributes["winlog.*"]</td>
</tr>
</table>
### SignalFx Events
<table>
<tr>
<td>Field</td>
<td>Type</td>
<td>Description</td>
<td>Maps to Unified Model Field</td>
</tr>
<tr>
<td>Timestamp</td>
<td>Timestamp</td>
<td>Time when the event occurred measured by the origin clock.</td>
<td>Timestamp</td>
</tr>
<tr>
<td>EventType</td>
<td>string</td>
<td>Short machine understandable string describing the event type. SignalFx specific concept. Non-namespaced. Example: k8s Event Reason field.</td>
<td>Attributes["com.splunk.signalfx.event_type"]</td>
</tr>
<tr>
<td>Category</td>
<td>enum</td>
<td>Describes where the event originated and why. SignalFx specific concept. Example: AGENT. </td>
<td>Attributes["com.splunk.signalfx.event_category"]</td>
</tr>
<tr>
<td>Dimensions</td>
<td>map<string, string></td>
<td>Helps to define the identity of the event source together with EventType and Category. Multiple occurrences of events coming from the same event source can happen across time and they all have the value of Dimensions. </td>
<td>Resource</td>
</tr>
<tr>
<td>Properties</td>
<td>map<string, any></td>
<td>Additional information about the specific event occurrence. Unlike Dimensions which are fixed for a particular event source, Properties can have different values for each occurrence of the event coming from the same event source.</td>
<td>Attributes</td>
</tr>
</table>
### Splunk HEC
We apply this mapping from HEC to the unified model:
<table>
<tr>
<td>Field</td>
<td>Type</td>
<td>Description</td>
<td>Maps to Unified Model Field</td>
</tr>
<tr>
<td>time</td>
<td>numeric, string</td>
<td>The event time in epoch time format, in seconds.</td>
<td>Timestamp</td>
</tr>
<tr>
<td>host</td>
<td>string</td>
<td>The host value to assign to the event data. This is typically the host name of the client that you are sending data from.</td>
<td>Resource["host.name"]</td>
</tr>
<tr>
<td>source</td>
<td>string</td>
<td>The source value to assign to the event data. For example, if you are sending data from an app you are developing, you could set this key to the name of the app.</td>
<td>Resource["com.splunk.source"]</td>
</tr>
<tr>
<td>sourcetype</td>
<td>string</td>
<td>The sourcetype value to assign to the event data.</td>
<td>Resource["com.splunk.sourcetype"]</td>
</tr>
<tr>
<td>event</td>
<td>any</td>
<td>The JSON representation of the raw body of the event. It can be a string, number, string array, number array, JSON object, or a JSON array.</td>
<td>Body</td>
</tr>
<tr>
<td>fields</td>
<td>map<string, any></td>
<td>Specifies a JSON object that contains explicit custom fields.</td>
<td>Attributes</td>
</tr>
<tr>
<td>index</td>
<td>string</td>
<td>The name of the index by which the event data is to be indexed. The index you specify here must be within the list of allowed indexes if the token has the indexes parameter set.</td>
<td>Attributes["com.splunk.index"]</td>
</tr>
</table>
When mapping from the unified model to HEC, we apply this additional mapping:
<table>
<tr>
<td>Unified model element</td>
<td>Type</td>
<td>Description</td>
<td>Maps to HEC</td>
</tr>
<tr>
<td>SeverityText</td>
<td>string</td>
<td>The severity of the event as a human-readable string.</td>
<td>fields['otel.log.severity.text']</td>
</tr>
<tr>
<td>SeverityNumber</td>
<td>string</td>
<td>The severity of the event as a number.</td>
<td>fields['otel.log.severity.number']</td>
</tr>
<tr>
<td>Name</td>
<td>string</td>
<td>Short event identifier that does not contain varying parts.</td>
<td>fields['otel.log.name']</td>
</tr>
<tr>
<td>TraceId</td>
<td>string</td>
<td>Request trace id.</td>
<td>fields['trace_id']</td>
</tr>
<tr>
<td>SpanId</td>
<td>string</td>
<td>Request span id.</td>
<td>fields['span_id']</td>
</tr>
<tr>
<td>TraceFlags</td>
<td>string</td>
<td>W3C trace flags.</td>
<td>fields['trace_flags']</td>
</tr>
</table>
### Log4j
<table>
<tr>
<td>Field</td>
<td>Type</td>
<td>Description</td>
<td>Maps to Unified Model Field</td>
</tr>
<tr>
<td>Instant</td>
<td>Timestamp</td>
<td>Time when an event occurred measured by the origin clock.</td>
<td>Timestamp</td>
</tr>
<tr>
<td>Level</td>
<td>enum</td>
<td>Log level.</td>
<td>Severity</td>
</tr>
<tr>
<td>Message</td>
<td>string</td>
<td>Human readable message.</td>
<td>Body</td>
</tr>
<tr>
<td>All other fields</td>
<td>any</td>
<td>Structured data.</td>
<td>Attributes</td>
</tr>
</table>
### Zap
<table>
<tr>
<td>Field</td>
<td>Type</td>
<td>Description</td>
<td>Maps to Unified Model Field</td>
</tr>
<tr>
<td>ts</td>
<td>Timestamp</td>
<td>Time when an event occurred measured by the origin clock.</td>
<td>Timestamp</td>
</tr>
<tr>
<td>level</td>
<td>enum</td>
<td>Logging level.</td>
<td>Severity</td>
</tr>
<tr>
<td>caller</td>
<td>string</td>
<td>Calling function's filename and line number.
</td>
<td>Attributes, key=TBD</td>
</tr>
<tr>
<td>msg</td>
<td>string</td>
<td>Human readable message.</td>
<td>Body</td>
</tr>
<tr>
<td>All other fields</td>
<td>any</td>
<td>Structured data.</td>
<td>Attributes</td>
</tr>
</table>
### Apache HTTP Server access log
<table>
<tr>
<td>Field</td>
<td>Type</td>
<td>Description</td>
<td>Maps to Unified Model Field</td>
</tr>
<tr>
<td>%t</td>
<td>Timestamp</td>
<td>Time when an event occurred measured by the origin clock.</td>
<td>Timestamp</td>
</tr>
<tr>
<td>%a</td>
<td>string</td>
<td>Client IP</td>
<td>Attributes["net.peer.ip"]</td>
</tr>
<tr>
<td>%A</td>
<td>string</td>
<td>Server IP</td>
<td>Attributes["net.host.ip"]</td>
</tr>
<tr>
<td>%h</td>
<td>string</td>
<td>Remote hostname. </td>
<td>Attributes["net.peer.name"]</td>
</tr>
<tr>
<td>%m</td>
<td>string</td>
<td>The request method.</td>
<td>Attributes["http.method"]</td>
</tr>
<tr>
<td>%v,%p,%U,%q</td>
<td>string</td>
<td>Multiple fields that can be composed into URL.</td>
<td>Attributes["http.url"]</td>
</tr>
<tr>
<td>%>s</td>
<td>string</td>
<td>Response status.</td>
<td>Attributes["http.status_code"]</td>
</tr>
<tr>
<td>All other fields</td>
<td>any</td>
<td>Structured data.</td>
<td>Attributes, key=TBD</td>
</tr>
</table>
### CloudTrail Log Event
<table>
<tr>
<td>Field</td>
<td>Type</td>
<td>Description</td>
<td>Maps to Unified Model Field</td>
</tr>
<tr>
<td>eventTime</td>
<td>string</td>
<td>The date and time the request was made, in coordinated universal time (UTC).</td>
<td>Timestamp</td>
</tr>
<tr>
<td>eventSource</td>
<td>string</td>
<td>The service that the request was made to. This name is typically a short form of the service name without spaces plus .amazonaws.com.</td>
<td>Resource["service.name"]?</td>
</tr>
<tr>
<td>awsRegion</td>
<td>string</td>
<td>The AWS region that the request was made to, such as us-east-2.</td>
<td>Resource["cloud.region"]</td>
</tr>
<tr>
<td>sourceIPAddress</td>
<td>string</td>
<td>The IP address that the request was made from.</td>
<td>Resource["net.peer.ip"] or Resource["net.host.ip"]? TBD</td>
</tr>
<tr>
<td>errorCode</td>
<td>string</td>
<td>The AWS service error if the request returns an error.</td>
<td>Attributes["cloudtrail.error_code"]</td>
</tr>
<tr>
<td>errorMessage</td>
<td>string</td>
<td>If the request returns an error, the description of the error.</td>
<td>Body</td>
</tr>
<tr>
<td>All other fields</td>
<td>*</td>
<td></td>
<td>Attributes["cloudtrail.*"]</td>
</tr>
</table>
### Google Cloud Logging
Field | Type | Description | Maps to Unified Model Field
-----------------|--------------------| ------------------------------------------------------- | ---------------------------
timestamp | string | The time the event described by the log entry occurred. | Timestamp
resource | MonitoredResource | The monitored resource that produced this log entry. | Resource
log_name | string | The URL-encoded LOG_ID suffix of the log_name field identifies which log stream this entry belongs to. | Attributes["gcp.log_name"]
json_payload | google.protobuf.Struct | The log entry payload, represented as a structure that is expressed as a JSON object. | Body
proto_payload | google.protobuf.Any | The log entry payload, represented as a protocol buffer. | Body
text_payload | string | The log entry payload, represented as a Unicode string (UTF-8). | Body
severity | LogSeverity | The severity of the log entry. | Severity
trace | string | The trace associated with the log entry, if any. | TraceId
span_id | string | The span ID within the trace associated with the log entry. | SpanId
labels | map<string,string> | A set of user-defined (key, value) data that provides additional information about the log entry. | Attributes
http_request | HttpRequest | The HTTP request associated with the log entry, if any. | Attributes["gcp.http_request"]
All other fields | | | Attributes["gcp.*"]
## Elastic Common Schema
<table>
<tr>
<td>Field</td>
<td>Type</td>
<td>Description</td>
<td>Maps to Unified Model Field</td>
</tr>
<tr>
<td>@timestamp</td>
<td>datetime</td>
<td>Time the event was recorded</td>
<td>Timestamp</td>
</tr>
<tr>
<td>message</td>
<td>string</td>
<td>Any type of message</td>
<td>Body</td>
</tr>
<tr>
<td>labels</td>
<td>key/value</td>
<td>Arbitrary labels related to the event</td>
<td>Attributes[*]</td>
</tr>
<tr>
<td>tags</td>
<td>array of string</td>
<td>List of values related to the event</td>
<td>?</td>
</tr>
<tr>
<td>trace.id</td>
<td>string</td>
<td>Trace ID</td>
<td>TraceId</td>
</tr>
<tr>
<td>span.id*</td>