Improve output when --full=false

documentation/functions/domain/IGNORE.md

@@ -4,15 +4,17 @@ parameters:
     - labelSpec
     - typeSpec
     - targetSpec
+    - silent
 parameter_types:
     labelSpec: string
     typeSpec: string?
     targetSpec: string?
+    silent: bool?
 ---
 
 `IGNORE()` makes it possible for DNSControl to share management of a domain
 with an external system.  The parameters of `IGNORE()` indicate which records
-are managed elsewhere and should not be modified or deleted.
+are managed elsewhere and should not be modified or deleted by DNSControl.
 
 Use case: Suppose a domain is managed by both DNSControl and a third-party
 system. This creates a problem because DNSControl will try to delete records
@@ -21,32 +23,35 @@ those records.  The two systems will get into an endless update cycle where
 each will revert changes made by the other in an endless loop.
 
 To solve this problem simply include `IGNORE()` statements that identify which
-records are managed elsewhere.  DNSControl will not modify or delete those
+records (or patterns) are managed elsewhere.  DNSControl will not modify or delete those
 records.
 
-Technically `IGNORE_NAME` is a promise that DNSControl will not modify or
-delete existing records that match particular patterns. It is like
-[`NO_PURGE`](../domain/NO_PURGE.md) that matches only specific records.
+Technically `IGNORE` is a promise that DNSControl will not modify or
+delete existing records that match particular patterns. It is similar to
+[`NO_PURGE`](../domain/NO_PURGE.md) but it matches only specific records.
 
-Including a record that is ignored is considered an error and may have
-undefined behavior. This safety check can be disabled using the
-[`DISABLE_IGNORE_SAFETY_CHECK`](../domain/DISABLE_IGNORE_SAFETY_CHECK.md) feature.
+It is considered to be an error if an`IGNORE()` pattern matches records in
+`dnsconfig.js`. This safety check can be disabled using the
+[`DISABLE_IGNORE_SAFETY_CHECK`](../domain/DISABLE_IGNORE_SAFETY_CHECK.md)
+feature. Behavior is undefined when the safety check is disabled.
 
 ## Syntax
 
-The `IGNORE()` function can be used with up to 3 parameters:
+The `IGNORE()` function can be used with up to 4 parameters:
 
 {% code %}
 ```javascript
+IGNORE(labelSpec, typeSpec, targetSpec, silent):
 IGNORE(labelSpec, typeSpec, targetSpec):
 IGNORE(labelSpec, typeSpec):
 IGNORE(labelSpec):
 ```
 {% endcode %}
 
 * `labelSpec` is a glob that matches the DNS label. For example `"foo"` or `"foo*"`.  `"*"` matches all labels, as does the empty string (`""`).
-* `typeSpec` is a comma-separated list of DNS types.  For example `"A"` matches DNS A records, `"A,CNAME"` matches both A and CNAME records. `"*"` matches any DNS type, as does the empty string (`""`).  
+* `typeSpec` is a comma-separated list of DNS types.  For example `"A"` matches DNS A records, `"A,CNAME"` matches both A and CNAME records. `"*"` matches any DNS type, as does the empty string (`""`).
 * `targetSpec` is a glob that matches the DNS target. For example `"foo"` or `"foo*"`.  `"*"` matches all targets, as does the empty string (`""`).
+* `silent` is a bool that, when set to `true`, indicates that records ignored using this rule do not need to be listed in `preview` and `push` when they announce ignored records.  Use the `--full` flag to show all records. If two or more `IGNORE()` statements match the same record and have different `silent` settings, results are undefined.
 
 `typeSpec` and `targetSpec` default to `"*"` if they are omitted.
 
@@ -64,10 +69,16 @@ following patterns will work:
 * `IGNORE("{bar,[fz]oo}")` will ignore `bar`, `foo` and `zoo`.
 * `IGNORE("\\*.foo")` will ignore the literal record `*.foo`.
 
+{% hint style="info" %}
+**NOTE**: `.` should not be escaped with a `\`.  These are globs (like filenames), not regular expressions.
+{% endhint %}
+
 ## Typical Usage
 
 General examples:
 
+Here we should typical usage.
+
 {% code title="dnsconfig.js" %}
 ```javascript
 D("example.com", REG_MY_PROVIDER, DnsProvider(DSP_MY_PROVIDER),
@@ -77,7 +88,11 @@ D("example.com", REG_MY_PROVIDER, DnsProvider(DSP_MY_PROVIDER),
   IGNORE("*", "CNAME", "dev-*"), // matches CNAMEs with targets prefixed `dev-*`
   IGNORE("bar", "A,MX"), // ignore only A and MX records for name bar
   IGNORE("*", "*", "dev-*"), // Ignore targets with a `dev-` prefix
-  IGNORE("*", "A", "1\.2\.3\."), // Ignore targets in the 1.2.3.0/24 CIDR block
+  IGNORE("*", "A", "10.2.3.*"), // Ignore targets in the 10.2.3.0/24 CIDR block
+  IGNORE("*", "A", "10.2.3.*", true), // Same as previous line, but records will not be reported unless `--full`
+  IGNORE("*", "A", "10.2.*"), // Ignore targets in the 10.2.0.0/16 CIDR block
+  IGNORE("foo", "", "", true), // Same as first example, with `silent` set to `true`.
+  IGNORE("foo", "*", "*", true), // Equivalent to the previous line.
 END);
 ```
 {% endcode %}
@@ -285,7 +300,7 @@ D("example.com", REG_MY_PROVIDER, DnsProvider(DSP_MY_PROVIDER),
 
 {% hint style="info" %}
 FYI: Previously DNSControl permitted disabling this check on
-a per-record basis using `IGNORE_NAME_DISABLE_SAFETY_CHECK`:
+a per-record basis using `IGNORE_NAME_DISABLE_SAFETY_CHECK`. This is now a domain-level setting.
 {% endhint %}
 
 The `IGNORE_NAME_DISABLE_SAFETY_CHECK` feature does not exist in the diff2
@@ -309,4 +324,4 @@ as a last resort. Even then, test extensively.
 * There is no locking.  If the external system and DNSControl make updates at the exact same time, the results are undefined.
 * IGNORE` works fine with records inserted into a `D()` via `D_EXTEND()`. The matching is done on the resulting FQDN of the label or target.
 * `targetSpec` does not match fields other than the primary target.  For example, `MX` records have a target hostname plus a priority. There is no way to match the priority.
-* The BIND provider can not ignore records it doesn't know about.  If it does not have access to an existing zonefile, it will create a zonefile from scratch. That new zonefile will not have any external records.  It will seem like they were not ignored, but in reality BIND didn't have visibility to them so that they could be ignored.
+* The BIND provider can not ignore records it doesn't know about.  If it does not have access to an existing zonefile, it will create a zonefile from scratch. That new zonefile will not have any external-created records.  It will seem like they were not ignored, but in reality BIND will simply create a new zone without those records.

models/record.go

@@ -130,6 +130,7 @@ type RecordConfig struct {
 	TxtStrings       []string          `json:"txtstrings,omitempty"` // TxtStrings stores all strings (including the first). Target stores all the strings joined.
 	R53Alias         map[string]string `json:"r53_alias,omitempty"`
 	AzureAlias       map[string]string `json:"azure_alias,omitempty"`
+	SilenceReporting bool              `json:"silence_reporting,omitempty"`
 }
 
 // MarshalJSON marshals RecordConfig.

models/unmanaged.go

@@ -26,6 +26,9 @@ type UnmanagedConfig struct {
 	// Glob pattern for matching targets.
 	TargetPattern string    `json:"target_pattern,omitempty"`
 	TargetGlob    glob.Glob `json:"-"` // Compiled version
+
+	// Output warnings about this ignore?
+	SilenceReporting bool `json:"silence_reporting,omitempty"`
 }
 
 // DebugUnmanagedConfig returns a string version of an []*UnmanagedConfig for debugging purposes.

pkg/diff2/handsoff.go

@@ -98,7 +98,7 @@ The actual implementation combines this all into one loop:
     Append "foreign list" to "desired".
 */
 
-const maxReport = 5
+const defaultMaxReport = 5
 
 // handsoff processes the IGNORE*()//NO_PURGE/ENSURE_ABSENT features.
 func handsoff(
@@ -118,19 +118,13 @@ func handsoff(
 
 	// Process IGNORE*() and NO_PURGE features:
 	ignorable, foreign := processIgnoreAndNoPurge(domain, existing, desired, absences, unmanagedConfigs, noPurge)
-	if len(foreign) != 0 {
-		msgs = append(msgs, fmt.Sprintf("%d records not being deleted because of NO_PURGE:", len(foreign)))
-		msgs = append(msgs, reportSkips(foreign, !printer.SkinnyReport)...)
-	}
-	if len(ignorable) != 0 {
-		msgs = append(msgs, fmt.Sprintf("%d records not being deleted because of IGNORE*():", len(ignorable)))
-		msgs = append(msgs, reportSkips(ignorable, !printer.SkinnyReport)...)
-	}
+	msgs = append(msgs, genReport(foreign, "NO_PURGE", defaultMaxReport)...)
+	msgs = append(msgs, genReport(ignorable, "IGNORE()", defaultMaxReport)...)
 
 	// Check for invalid use of IGNORE_*.
 	conflicts := findConflicts(unmanagedConfigs, desired)
 	if len(conflicts) != 0 {
-		msgs = append(msgs, fmt.Sprintf("%d records that are both IGNORE*()'d and not ignored:", len(conflicts)))
+		msgs = append(msgs, fmt.Sprintf("%d records that are both IGNORE()'d and not ignored:", len(conflicts)))
 		for _, r := range conflicts {
 			msgs = append(msgs, fmt.Sprintf("    %s %s %s", r.GetLabelFQDN(), r.Type, r.GetTargetCombined()))
 		}
@@ -146,47 +140,129 @@ func handsoff(
 	return desired, msgs, nil
 }
 
-// reportSkips reports records being skipped, if !full only the first maxReport are output.
-func reportSkips(recs models.Records, full bool) []string {
-	var msgs []string
+// genReport generates a report of what records were not deleted with a human-readable header and footer.  Abides by maxReport.
+func genReport(recs models.Records, reason string, maxReport int) (msgs []string) {
+	if len(recs) == 0 {
+		return nil
+	}
+	visibleCount, hiddenCount := countVisibility(recs)
+	header, footer := makeHeaderFooter(reason, !printer.SkinnyReport, maxReport, visibleCount, hiddenCount)
+	msgs = append(msgs, header)
+	msgs = append(msgs, reportMessages(recs, maxReport, !printer.SkinnyReport)...)
+	if footer != "" {
+		msgs = append(msgs, footer)
+	}
+
+	return msgs
+}
+
+// makeHeaderFooter generates fancy header and footer.
+func makeHeaderFooter(reason string, full bool, maxReport, visibleCount, hiddenCount int) (header, footer string) {
+
+	totalCount := visibleCount + hiddenCount
+
+	// With the --full flag, use a very simple header and no footer.
+	if full {
+		header = fmt.Sprintf("FYI: %d records NOT deleted due to %s:", totalCount, reason)
+		footer = ""
+		return header, footer
+	}
+
+	shownCount := visibleCount
+	if visibleCount > maxReport {
+		shownCount = maxReport
+	}
+
+	punct := ":"
+	// If no records will be output, change the punctuation at the end to "."
+	if shownCount == 0 {
+		punct = "."
+	}
+
+	header = fmt.Sprintf("%d records NOT deleted due to %s%s (%d/%d/%d displayed/visible/hidden)",
+		totalCount, reason, punct,
+		shownCount, visibleCount, hiddenCount,
+	)
+	if (totalCount != shownCount) || (hiddenCount != 0) {
+		// Only add this if adding the flag would change the output.
+		header = header + " (--full shows all)"
+	}
+
+	// The footer visually indicates that we've hit the maxReport limit.
+	footer = ""
+	if visibleCount != shownCount {
+		footer = "     ..."
+	}
+
+	return header, footer
+}
 
-	shorten := (!full) && (len(recs) > maxReport)
-	last := len(recs)
-	if shorten {
-		last = maxReport
+// countVisibility returns how many records are visible/hidden.
+func countVisibility(recs models.Records) (visibleCount, hiddenCount int) {
+	for _, r := range recs {
+		if r.SilenceReporting {
+			hiddenCount++
+		} else {
+			visibleCount++
+		}
 	}
+	return visibleCount, hiddenCount
+}
 
-	for _, r := range recs[:last] {
-		msgs = append(msgs, fmt.Sprintf("    %s. %s %s", r.GetLabelFQDN(), r.Type, r.GetTargetCombined()))
+// reportMessages generates one message for each record, abiding by maxReport limits.
+func reportMessages(recs models.Records, maxReport int, full bool) (msgs []string) {
+	if len(recs) == 0 {
+		return nil
 	}
-	if shorten {
-		msgs = append(msgs, fmt.Sprintf("    ...and %d more... (use --full to show all)", len(recs)-maxReport))
+
+	if full {
+		for _, r := range recs {
+			msgs = append(msgs, genRecordMessage(r))
+		}
+		return msgs
+	}
+
+	for _, r := range recs {
+		if !r.SilenceReporting {
+			msgs = append(msgs, genRecordMessage(r))
+			if len(msgs) == maxReport {
+				break
+			}
+		}
 	}
 
 	return msgs
 }
 
+// genRecordMessage generate the message for one record.
+func genRecordMessage(r *models.RecordConfig) string {
+	return fmt.Sprintf("    %s. %s %s", r.GetLabelFQDN(), r.Type, r.GetTargetCombined())
+}
+
 // processIgnoreAndNoPurge processes the IGNORE_*() and NO_PURGE/ENSURE_ABSENT() features.
 func processIgnoreAndNoPurge(domain string, existing, desired, absences models.Records, unmanagedConfigs []*models.UnmanagedConfig, noPurge bool) (models.Records, models.Records) {
 	var ignorable, foreign models.Records
 	desiredDB := models.NewRecordDBFromRecords(desired, domain)
 	absentDB := models.NewRecordDBFromRecords(absences, domain)
 	compileUnmanagedConfigs(unmanagedConfigs)
 	for _, rec := range existing {
-		isMatch := matchAny(unmanagedConfigs, rec)
+		isMatch, silence := matchAny(unmanagedConfigs, rec)
 		//fmt.Printf("DEBUG: matchAny returned: %v\n", isMatch)
-		if isMatch {
-			ignorable = append(ignorable, rec)
-		} else {
-			if noPurge {
-				// Is this a candidate for purging?
-				if !desiredDB.ContainsLT(rec) {
-					// Yes, but not if it is an exception!
-					if !absentDB.ContainsLT(rec) {
-						foreign = append(foreign, rec)
-					}
+		if isMatch && silence { // Marked as silent?
+			rec.SilenceReporting = true
+		}
+
+		if noPurge { // Process NO_PURGE.
+			// Is this a candidate for purging?
+			if !desiredDB.ContainsLT(rec) {
+				// Yes, but not if it is an exception!
+				if !absentDB.ContainsLT(rec) {
+					foreign = append(foreign, rec)
 				}
 			}
+			//}
+		} else if isMatch { // Process IGNORE() matches.
+			ignorable = append(ignorable, rec)
 		}
 	}
 	return ignorable, foreign
@@ -197,7 +273,7 @@ func processIgnoreAndNoPurge(domain string, existing, desired, absences models.R
 func findConflicts(uconfigs []*models.UnmanagedConfig, recs models.Records) models.Records {
 	var conflicts models.Records
 	for _, rec := range recs {
-		if matchAny(uconfigs, rec) {
+		if ans, _ := matchAny(uconfigs, rec); ans {
 			conflicts = append(conflicts, rec)
 		}
 	}
@@ -241,16 +317,16 @@ func compileUnmanagedConfigs(configs []*models.UnmanagedConfig) error {
 }
 
 // matchAny returns true if rec matches any of the uconfigs.
-func matchAny(uconfigs []*models.UnmanagedConfig, rec *models.RecordConfig) bool {
+func matchAny(uconfigs []*models.UnmanagedConfig, rec *models.RecordConfig) (bool, bool) {
 	//fmt.Printf("DEBUG: matchAny(%s, %q, %q, %q)\n", models.DebugUnmanagedConfig(uconfigs), rec.NameFQDN, rec.Type, rec.GetTargetField())
 	for _, uc := range uconfigs {
 		if matchLabel(uc.LabelGlob, rec.GetLabel()) &&
 			matchType(uc.RTypeMap, rec.Type) &&
 			matchTarget(uc.TargetGlob, rec.GetTargetField()) {
-			return true
+			return true, uc.SilenceReporting
 		}
 	}
-	return false
+	return false, false
 }
 func matchLabel(labelGlob glob.Glob, labelName string) bool {
 	if labelGlob == nil {

pkg/diff2/handsoff_test.go

@@ -234,3 +234,38 @@ _2222222222222222.cr CNAME _333333.nnn.acm-validations.aws.
 FOREIGN:
 	`)
 }
+
+func Test_reportSkips(t *testing.T) {
+
+	yes := &models.RecordConfig{Type: "A", SilenceReporting: true}
+	no := &models.RecordConfig{Type: "A", SilenceReporting: false}
+	maxReport := 5
+
+	tests := []struct {
+		name         string
+		data         models.Records
+		countNotFull int
+	}{
+		{"no2", models.Records{no, no}, 2},
+		{"yes2", models.Records{yes, yes}, 0},
+		{"noMax", models.Records{no, no, no, no, no}, maxReport},
+		{"no6", models.Records{no, no, no, no, no, no}, maxReport},
+		{"noMaxyes2", models.Records{no, no, no, no, no, yes, yes}, maxReport},
+		{"no6yes2", models.Records{no, no, no, no, no, no, yes, yes}, maxReport},
+		{"no6yes2mixed", models.Records{yes, no, no, no, yes, no, no, no}, maxReport},
+		{"yes8", models.Records{yes, yes, yes, yes, yes, yes, yes, yes}, 0},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			// If full==true, there should be 1 msg for each item in tt.data:
+			if got := reportMessages(tt.data, maxReport, true); len(got) != len(tt.data) {
+				t.Errorf("reportSkips(~, %d, true) = %v, want %v", len(got), maxReport, len(tt.data))
+			}
+			// If full==false, there should be 1 msg for the first maxReport no's:
+			if got := reportMessages(tt.data, maxReport, false); len(got) != tt.countNotFull {
+				fmt.Printf("MSGS=%v\n", strings.Join(got, ":\n"))
+				t.Errorf("reportSkips(-, false) = %v, want %v", len(got), tt.countNotFull)
+			}
+		})
+	}
+}